InferenceLocal Inference

Choose a Local Inference Server

View as Markdown

NemoClaw supports Ollama, vLLM, and NVIDIA NIM as local inference servers. Choose the option that matches your host, model, and operational needs.

The agent inside the sandbox sends inference traffic to inference.local. OpenShell intercepts that traffic and forwards it to the local endpoint configured during onboarding.

Compare the Options

OptionWhen to use itAvailabilityRuntime API
Existing vLLMYou already run vLLM on localhost:8000.Appears when NemoClaw detects the server./v1/chat/completions.
Managed vLLMYou want NemoClaw to pull an image, download model weights, and manage the server container.Appears by default on DGX Spark and DGX Station, while generic Linux NVIDIA GPU hosts require NEMOCLAW_EXPERIMENTAL=1 or NEMOCLAW_PROVIDER=install-vllm./v1/chat/completions.
NVIDIA NIMYou want NemoClaw to pull and manage a validated NIM container on a NIM-capable NVIDIA GPU.Experimental and requires NEMOCLAW_EXPERIMENTAL=1./v1/chat/completions.

Managed vLLM uses host-specific model profiles and lets you select a supported registry model. NVIDIA NIM filters its available models by detected GPU VRAM.

Choose vLLM

Choose vLLM when you already operate a compatible server or want a managed container on a supported NVIDIA GPU host. NemoClaw forces the Chat Completions API path because the vLLM Responses endpoint does not run the configured tool-call parser.

Refer to Set Up vLLM.

Choose NVIDIA NIM

Choose NVIDIA NIM when you want a managed NIM container and your host has a NIM-capable NVIDIA GPU. The path is experimental, requires NGC registry access, and can fail when a selected image does not publish a manifest for the host architecture.

Refer to Set Up NVIDIA NIM.

Use Another Server

Use a custom endpoint when your server is not one of the managed local options. NemoClaw supports servers that expose an OpenAI-compatible API and supports compatible Anthropic routes with agent-specific runtime requirements.