Choose a Local Inference Server

NemoClaw supports operator-run servers and NemoClaw-managed runtime profiles for local inference. Choose the option that matches your host, model, and operational needs.

The agent inside the sandbox sends inference traffic to inference.local. OpenShell intercepts that traffic and forwards it to the local endpoint configured during onboarding.

Compare the Options

Option	When to use it	Availability	Runtime API
Existing vLLM	You already run vLLM on `localhost:8000`.	Appears when NemoClaw detects the server.	`/v1/chat/completions`.
Managed vLLM	You want NemoClaw to pull an image, download model weights, and manage the server container.	Appears by default on DGX Spark and DGX Station, while generic Linux NVIDIA GPU hosts require `NEMOCLAW_EXPERIMENTAL=1` or `NEMOCLAW_PROVIDER=install-vllm`.	`/v1/chat/completions`.
Fixed vLLM profile	You need the fixed catalog-selected vLLM model and serving configuration.	Requires the dedicated installer flag on DGX Spark and does not appear in the provider menu.	`/v1/chat/completions`.
Managed llama.cpp	You want NemoClaw to acquire a verified GGUF file and manage an authenticated llama.cpp container.	Experimental. Appears on a qualified DGX Spark host and supports explicit non-interactive selection.	`/v1/chat/completions`.
NVIDIA NIM	You want NemoClaw to pull and manage a validated NIM container on a NIM-capable NVIDIA GPU.	Experimental and requires `NEMOCLAW_EXPERIMENTAL=1`.	`/v1/chat/completions`.

Managed vLLM uses host-specific model profiles and lets you select a supported registry model. NVIDIA NIM filters its available models by detected GPU VRAM.

Choose vLLM

Choose vLLM when you already operate a compatible server or want a managed container on a supported NVIDIA GPU host. NemoClaw forces the Chat Completions API path because the vLLM Responses endpoint does not run the configured tool-call parser.

Refer to Set Up vLLM.

Install a Fixed vLLM Profile

Use this feature-gated path to install the fixed catalog-selected vLLM model and runtime combination. The installer does not accept a provider or model override for this profile.

Before you start, confirm these prerequisites:

Use a DGX Spark host with Linux on Arm64.
Confirm that Docker, the NVIDIA Container Toolkit, and the NVIDIA GPU driver are operational.
Stop any server that already uses port 8000.
Allow capacity for container images, model artifacts, and same-filesystem download staging.
Export HF_TOKEN only if the configured artifact source requires Hugging Face authentication.

NemoClaw uses HF_TOKEN only for model acquisition and does not write it to NemoClaw state. The value remains in the caller’s environment; run unset HF_TOKEN after installation when no other process needs it.

The installer downloads a pinned container image and a fixed catalog model from external registries. It runs the remaining onboarding steps non-interactively and can recreate the named sandbox when its recorded inference route differs. The runtime stores an owner-only bearer API key and model artifacts on the host.

$ curl -fsSL https://www.nvidia.com/nemoclaw.sh | \
>   NEMOCLAW_AGENT=langchain-deepagents-code \
>   NEMOCLAW_SANDBOX_NAME=my-assistant \
>   NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE=1 \
>   bash -s -- --local-model-runtime=vllm

The flag enables the dedicated vLLM profile gate, disables Express selection, and selects the matching catalog recipe. Do not combine this path with NEMOCLAW_PROVIDER or NEMOCLAW_MODEL. The profile rejects NEMOCLAW_VLLM_MODEL, NEMOCLAW_VLLM_PORT, and NEMOCLAW_VLLM_EXTRA_ARGS_JSON before installation.

The profile performs these actions:

Uses the pinned runtime image and fixed serving command from the catalog.
Stores model files in the host Hugging Face cache.
Publishes the authenticated server only on 127.0.0.1:8000.
Reuses the owner-only host-global managed-vLLM API key.

After the runtime passes its readiness check, onboarding registers the provider and routes sandbox traffic through inference.local.

Verify the sandbox route:

$ nemo-deepagents my-assistant status
$ nemo-deepagents my-assistant doctor

Accept the result when status reports the inference route as reachable and doctor exits with status 0. Route reachability does not by itself establish successful model invocation.

Verify the loopback publication:

$ docker port nemoclaw-vllm

The result must contain 127.0.0.1:8000.

If installation stops after a model download, rerun the same installer command. The runtime reuses only an artifact that passes its recorded identity checks. If Docker reports a container-name conflict, inspect the resource labels before removing anything. Do not remove a resource that lacks the NemoClaw ownership label for this profile. Refer to Host Files and State before deleting a credential or shared cached artifact.

When you intend to remove the entire NemoClaw installation, run nemo-deepagents uninstall. Full uninstall verifies exact managed container ownership before it removes the runtime and its state. It preserves the shared Hugging Face cache used by vLLM. Add --delete-models only when you also accept deletion of NemoClaw-pulled Ollama models. If cleanup reports an ownership or Docker error, leave the state files in place, resolve the reported resource, and rerun uninstall. After a successful full uninstall, docker container inspect nemoclaw-vllm must report no object.

Install Managed llama.cpp on DGX Spark

Use this experimental path when you want NemoClaw to manage one declarative llama.cpp recipe on one DGX Spark. The default recipe serves NVIDIA Nemotron 3 Nano 30B-A3B through the OpenAI Chat Completions API.

This path is an experimental implementation. It does not establish a supported agent, model, and runtime tuple until the protected qualification and activation gates pass.

Before you start, confirm these prerequisites:

Use a DGX Spark host with Linux on Arm64.
Confirm that Docker and the NVIDIA Container Toolkit are operational.
Confirm that NVIDIA Container Device Interface support is healthy.
Use NVIDIA driver version 580.65.06 or later.
Stop any process that uses host port 8081.
Allow capacity for the pinned images, the GGUF file, and same-filesystem download staging.
Export HF_TOKEN only when the YAML-declared Hugging Face source requires authentication.

NemoClaw uses HF_TOKEN only while it acquires the model. It does not write the value to managed llama.cpp state, the receipt, the serving container environment, or the sandbox registry. Run unset HF_TOKEN after onboarding when no other process needs it.

When the exact artifacts are not already present, onboarding pulls digest-pinned images from external registries and acquires the YAML-declared GGUF file through the standard Hugging Face cache. It reuses locally present pinned images and a cached GGUF only after their exact identities verify. It creates an owner-only API key, a host-loopback listener on port 8081, and a Docker internal network. Only one managed llama.cpp runtime can exist per Docker authority, regardless of the owning OpenShell gateway or sandbox.

Run interactive onboarding on the DGX Spark and select NVIDIA Nemotron with managed llama.cpp (DGX Spark):

$ nemo-deepagents onboard

For non-interactive onboarding, select the repository-owned recipe explicitly:

$ NEMOCLAW_PROVIDER=install-llama-cpp \
> NEMOCLAW_LLAMACPP_RECIPE=llama-cpp.nemotron-3-nano-30b-a3b.spark-single.v1 \
> NEMOCLAW_SANDBOX_NAME=my-assistant \
>   nemo-deepagents onboard --non-interactive --yes-i-accept-third-party-software

When NEMOCLAW_LLAMACPP_RECIPE is unset, NemoClaw selects the one shipped managed llama.cpp recipe. Do not set NEMOCLAW_MODEL for this path. The recipe owns the exact model file, revision, digest, image, launch settings, readiness probes, and disabled surfaces.

Onboarding performs these actions:

Resolves the recipe against a fresh DGX Spark readiness report.
Reuses locally present pinned downloader, runtime, and probe images, and pulls only the missing digest-pinned images declared by the recipe.
Reuses a verified GGUF from the shared ~/.cache/huggingface/ cache, or acquires the exact file through the existing Hugging Face mechanism when it is absent or invalid.
Verifies the GGUF size and SHA-256 digest before it starts the runtime.
Runs the authenticated container as the current non-root user with one NVIDIA GPU and no CPU fallback.
Publishes port 8081 on 127.0.0.1 and connects the container to a Docker internal network.
Registers llama-cpp-local and routes agent traffic through https://inference.local/v1.

The runtime cannot download a model and has no egress on its internal Docker network. The recipe disables the Web UI, slot inspection, server tools, agent mode, Model Context Protocol proxy, router, and multimodal projection. The API key enters the container through an owner-only read-only file, not a process argument.

Verify the managed runtime and sandbox route:

$ nemo-deepagents my-assistant status
$ nemo-deepagents my-assistant doctor
$ docker port nemoclaw-llama-cpp

Accept the result when status reports Managed llama.cpp: running, the inference route is reachable, and doctor exits with status 0. The Docker port output must contain 127.0.0.1:8081. These checks do not establish agent and model qualification.

If onboarding stops, rerun the same provider and recipe selection. NemoClaw resumes only the exact persisted Docker authority, image, recipe, model digest, network, and runtime identity. It reconciles an unfinished create journal before it starts a new runtime: phases before receipt-prepared are rolled back and retired, while receipt-prepared is finalized.

If status reports stopped or preparing, rerun the same onboarding selection to resume the runtime. If it reports absent, conflict, or unknown, inspect the reported identity or Docker-authority error before retrying. Do not remove a same-name container or network unless its labels and identifiers match the persisted ownership state.

nemo-deepagents my-assistant destroy removes the exact managed llama.cpp container, internal network, API key, and gateway-scoped ownership state after OpenShell confirms sandbox deletion. It preserves ~/.cache/huggingface/ because other applications can use that cache. If exact cleanup fails, NemoClaw preserves the sandbox registry entry and ownership state for a retry.

Choose NVIDIA NIM

Choose NVIDIA NIM when you want a managed NIM container and your host has a NIM-capable NVIDIA GPU. The path is experimental, requires NGC registry access, and can fail when a selected image does not publish a manifest for the host architecture.

Refer to Set Up NVIDIA NIM.

Use Another Server

Use a custom endpoint when your server is not one of the managed local options. NemoClaw supports servers that expose an OpenAI-compatible API and supports compatible Anthropic routes with agent-specific runtime requirements.

Configure Inference Timeouts for slow local models and long sandbox startup times.
Verify the Inference Route after onboarding.