Set Up vLLM
NemoClaw can connect to an existing vLLM server or install and start a managed vLLM container on supported NVIDIA GPU hosts.
NemoClaw forces the Chat Completions API path for vLLM.
The vLLM /v1/responses endpoint does not run the configured tool-call parser, so tool calls can arrive as raw text.
Local vLLM does not require authentication by default.
Existing-server and single-host managed-vLLM paths need port 8000 on host loopback for validation and on the OpenShell Docker bridge for sandbox traffic.
Use a host firewall with default-deny inbound rules.
Allow TCP port 8000 only from the OpenShell Docker subnet to its gateway address, keep loopback access, and deny the port on every other interface.
Do not expose the port to your LAN or the internet.
The qualified two-DGX Spark and two-DGX Station paths use bearer-protected /v1 routes and Docker host networking on both runtime containers; network isolation remains required for other endpoints and distributed traffic.
Follow Set Up vLLM on Two DGX Sparks or Set Up vLLM on Two DGX Stations for their isolation requirements.
Use an Existing Server
When vLLM is already running on localhost:8000, NemoClaw detects it and queries /v1/models for the loaded model.
Run onboarding.
Select Local vLLM.
On generic hosts, the entry includes an experimental label; on DGX Spark or DGX Station, it does not.
NemoClaw validates the detected endpoint and records the model reported by /v1/models.
When an install request encounters an already-running server, NemoClaw selects that server instead of starting another container.
If the requested registry model and the served id differ, NemoClaw accepts the served id only when /v1/models reports a safe root that exactly matches the registry model.
NemoClaw rejects mismatched or unverifiable roots, and it still requires the exact served identity for a newly started NemoClaw-managed container and for an existing shared-gateway route.
When vLLM exposes runtime metadata such as max_model_len, NemoClaw uses that value for the configured context window unless you set NEMOCLAW_CONTEXT_WINDOW.
Start vLLM with the model you want before onboarding because NemoClaw uses the model already served by the endpoint.
On DGX Spark, an existing vLLM server is operator-managed.
NemoClaw warns when vLLM metadata heuristically indicates a large checkpoint without reported quantization, does not identify the underlying model size, or reports an unusually long context window.
These configurations can leave too little unified-memory headroom for the host and surface as NVRM NV_ERR_NO_MEMORY, SSH loss, or a hard host freeze under agent/tool load.
Prefer the managed Spark vLLM path when you want NemoClaw to use the validated Spark recipe, or restart your existing server with lower --gpu-memory-utilization, --max-model-len, --max-num-seqs, and --max-num-batched-tokens before onboarding.
For full diagnostics and recovery steps, refer to DGX Spark troubleshooting.
Configure Structured Tool Calls
Start vLLM with automatic tool choice and the tool-call parser that matches the model family when the model serves a tool-using agent. For a Hermes 3 style model, use this command shape.
Use the parser required by your selected model instead of copying hermes for another model family.
Without both --enable-auto-tool-choice and a matching parser, tool calls can appear as raw assistant text instead of structured tool_calls.
Install or Start Managed vLLM
NemoClaw displays the Install vLLM or Start vLLM entry by default on DGX Spark and DGX Station.
Generic Linux NVIDIA GPU hosts require NEMOCLAW_EXPERIMENTAL=1 or NEMOCLAW_PROVIDER=install-vllm before the managed entry appears.
Discover Serving Profiles
List the serving profiles installed with NemoClaw before you select a managed inference configuration.
The command evaluates the catalog against the current host without downloading a model or changing host, gateway, inference, or sandbox resources.
Each entry includes a stable ID, display name, backend, model, topology, selection mode, support state, estimated downloads, and an incompatibility reason when applicable.
Add --json when a script needs the same fields as structured output.
Select a Serving Profile
Pass a compatible profile’s stable ID to onboarding.
The same profile selector works with interactive and non-interactive onboarding.
NemoClaw rejects unknown, ambiguous, disabled, incompatible, or conflicting selections before image or model downloads begin.
Before confirmation, the review screen shows the resolved profile and recipe IDs, model, immutable runtime image, support state, and estimated image and model downloads.
Do not combine --profile with NEMOCLAW_PROVIDER, NEMOCLAW_MODEL, NEMOCLAW_VLLM_MODEL, NEMOCLAW_MANAGED_CLUSTER_PEERS, or NEMOCLAW_VLLM_EXTRA_ARGS_JSON overrides.
If NEMOCLAW_SERVING_PRESET is already set, it must select the same stable profile ID; a different ID conflicts with --profile.
The profile selector is model-independent, so a catalog can add profiles without adding model-specific CLI flags.
Omitting --profile preserves the existing provider and model defaults.
If onboarding is interrupted, nemo-deepagents onboard --resume reuses the recorded catalog, preset, and recipe digests.
It exits before runtime effects if the installed catalog changed instead of silently switching the recipe; use nemo-deepagents onboard --fresh --profile <profile-id> to review and start with the current definition.
Legacy sessions that predate profile provenance still resume with their original behavior, but you cannot add --profile to one of those in-progress sessions.
In an interactive run, the managed path lists the supported model registry for your host profile.
Press Enter to use the profile default or select another validated model with its registered vllm serve arguments.
The single-host path pulls the vLLM image, downloads model weights into ~/.cache/huggingface, and starts the nemoclaw-vllm container on localhost:8000.
The distributed DGX pages document their separate container names, endpoints, and lifecycle checks.
The container carries NemoClaw ownership and immutable catalog labels so later onboarding, recovery, and uninstall runs can distinguish it from a bring-your-own server without trusting the container name alone.
NemoClaw also records an owner-only, credential-free runtime receipt that binds those labels to the exact container ID and API-key fingerprint.
If an existing nemoclaw-vllm container does not carry that label, NemoClaw preserves it and asks you to remove or rename it before installing managed vLLM.
If another vLLM server already occupies port 8000, NemoClaw selects the existing-server path and applies the model-identity checks described above.
Managed profiles and model-specific recipes use immutable image digests:
- DGX Spark and DGX Station models without a model-specific runtime use the
linux/arm64digestsha256:9204569b17ee4c0eff75194b8e6e458479c8aee18953b5ab9cf359fcdac659e2with a compressed layer size of9.60 GBundernvcr.io/nvidia/vllm:26.05.post1-py3. - The DGX Station Nemotron 3 Ultra express recipe uses the multi-platform index digest
sha256:0fec7ec5f3e6bc168e54899935fb0557da908a4832a1dbc88e2debcf2f889416undervllm/vllm-openai:v0.22.0; on DGX Station, that index selects alinux/arm64manifest with a compressed layer size of10.67 GB. It also pins Hugging Face revision183968f87ae4cedce3039313cac1fd43d112c578for the approximately352.38 GBmodel download. - Generic Linux
arm64hosts usesha256:447995cbb57e6c7cf792cab95e9852e5f62b5fb6d2f39e030fa4eda9a54eadb4with a compressed layer size of9.28 GBundernvcr.io/nvidia/vllm:26.03.post1-py3. - Generic Linux
amd64hosts usesha256:7be6c2f676c36059a494fe17254e69ae5c677535ba6191044e5fc8e42a91c773with a compressed layer size of8.93 GBundernvcr.io/nvidia/vllm:26.03.post1-py3.
Image pulls from nvcr.io require NGC registry login.
The Station Nemotron 3 Ultra recipe pulls its vLLM image from Docker Hub instead.
The two-Station page records the distributed recipe’s image and runtime versions.
The first run duration depends on image and model size, storage performance, and network throughput.
Later runs reuse the cached image and model weights.
The single-host managed container uses Docker’s --restart unless-stopped policy, so Docker restarts it unless an operator explicitly stops it.
Authenticate Hugging Face Downloads
Authentication is optional for public models, but a read token reduces exposure to anonymous HTTP 429 rate limiting during large downloads.
Create a read token at Hugging Face Access Tokens, then export it in your shell before you start or restart onboarding.
Do not paste the token into NemoClaw prompts or chat.
NemoClaw reports whether the download is authenticated without displaying the token value.
It passes the token only to the temporary model downloader (hf download) and does not retain it in onboarding state or the long-running vLLM container.
NemoClaw also accepts HUGGING_FACE_HUB_TOKEN as a compatibility alias, but new setup should use HF_TOKEN.
When neither variable is present, public-model downloads continue anonymously.
If hf download reports HTTP 429, export HF_TOKEN and resume onboarding.
The downloader reuses files already stored in ~/.cache/huggingface instead of starting the model download from an empty cache.
Authentication reduces anonymous rate-limit exposure but does not guarantee that a download cannot receive HTTP 429.
Gated models still require license acceptance and a token before NemoClaw starts image or model downloads.
Check GPU Compute Capability
A quantized checkpoint does not load on a GPU whose compute capability is lower than the capability the checkpoint was built for.
Before the image pull and the model download, NemoClaw reads the compute capability of each GPU that nvidia-smi reports.
It compares the lowest reported capability against the selected model’s minimum.
On a mismatch, the managed install stops before those downloads and the error names both values.
Interactive onboarding returns to the inference-provider list so you can select another provider or model. Non-interactive onboarding exits.
Select a Managed Model lists the minimum for each registry entry.
A registry entry with no declared minimum, such as an unquantized checkpoint, skips this check.
When nvidia-smi reports no compute capability, NemoClaw continues and leaves the decision to vLLM.
Check Storage Before Downloads
For a cold managed-vLLM install, NemoClaw estimates local storage before the watched image pull starts.
The estimate adds the pinned image’s compressed layer size, the digest catalog’s pre-measured unpacked layer size when available, the selected model’s Hugging Face file size, a 3 GiB model-download staging reserve, and a writable allowance.
For the DGX Station default, that is approximately 393.68 GB: 9.603 GB compressed image layers, 27.659 GB unpacked image layers, 352.381 GB model files, 3.221 GB model-download staging, and 0.816 GB writable allowance.
For Docker’s classic image store, NemoClaw checks the absolute DockerRootDir reported by docker info.
For Docker’s containerd image store, it checks the top-level root from /etc/containerd/config.toml, or /var/lib/containerd when that configuration file is absent, and also checks DockerRootDir for pull staging.
For model files, NemoClaw checks the filesystem backing ~/.cache/huggingface.
The check compares Docker image storage against the image bytes and compares model-cache storage against the model files, download staging, and writable allowance.
When those paths share a filesystem, NemoClaw compares the shared capacity against the aggregate requirement.
When NemoClaw cannot determine whether successful checks share a filesystem, it conservatively compares their aggregate requirement against the lowest reported capacity.
NemoClaw does not move the Hugging Face cache to /raid or another fallback mount during this preflight.
NemoClaw recognizes an image cache hit only when Docker contains the exact platform digest.
A cache hit skips the image portion of the estimate, but still checks the model-cache estimate before hf download.
The explicit watched docker pull still runs against that immutable digest.
After an image pull, NemoClaw checks the model-cache estimate again before hf download.
The model-download helper and the long-lived vLLM container use --pull=never, so an implicit pull cannot select another image.
NemoClaw treats /var/run/docker.sock and /run/docker.sock as the local default Docker socket paths.
It cannot inspect daemon storage through a remote endpoint, a named non-default Docker context, or another Unix socket.
When both DOCKER_CONTEXT and DOCKER_HOST are set, NemoClaw follows Docker CLI precedence and uses DOCKER_CONTEXT.
Before a registered model with pinned size metadata is downloaded, NemoClaw also checks the filesystem that contains the host Hugging Face cache.
When a Docker image-storage check is inconclusive, NemoClaw prints the reason, approximate requirement, and diagnostic commands, then continues without claiming that capacity is sufficient. An inconclusive model-cache check remains guarded because a registered model download can be hundreds of gigabytes. For an inconclusive model-cache check, interactive setup requires explicit confirmation, and non-interactive setup stops with guidance to rerun interactively.
When verified capacity is insufficient, the warning identifies the image, model, available capacity, approximate requirement, checked path and source, remediation, and diagnostic commands.
Interactive setup continues only when you enter y or yes explicitly.
Pressing Enter, entering n or no, or entering any other response stops before the guarded image or model download starts.
--yes and NEMOCLAW_YES=1 do not change managed vLLM storage-warning handling.
Express installation and other non-interactive setup treat a verified warning that Docker image storage or model-cache storage is insufficient as advisory, so NemoClaw prints the warning and continues the guarded download.
Interactive setup still requires you to enter y or yes for a verified insufficient-capacity warning.
An inconclusive model-cache warning also requires explicit confirmation interactively, and non-interactive setup stops with guidance to rerun interactively. An inconclusive Docker image-storage check retains its existing automatic-continue behavior.
The model-cache guard applies when the selected registered recipe declares a pinned snapshot size.
NemoClaw creates ~/.cache/huggingface as the host user and runs the one-shot download container with the host UID and GID, avoiding root-owned cache paths.
Hugging Face credentials are forwarded only to that temporary downloader, not retained by the long-lived vLLM container.
A legacy root-owned or otherwise unwritable required model-cache path stops before the image pull and prints a scoped ownership-repair command; NemoClaw never changes its ownership automatically.
If image pull output stops making progress, a watchdog stops the stalled pull without imposing a fixed wall-clock limit on slow but active downloads.
If vLLM does not become ready, NemoClaw prints a short tail of the container logs before exiting.
Because the single-host managed container carries the --restart unless-stopped policy described above, a container that cannot serve its model restarts instead of exiting.
While NemoClaw waits for the API, it also reads that container’s restart count.
After the container restarts three times, NemoClaw stops the container and the install instead of waiting out the full load timeout.
When you start managed vLLM outside the installer express flow, NemoClaw uses these profile defaults:
For the managed single-node DGX Spark nvidia/Qwen3.6-35B-A3B-NVFP4 profile, NemoClaw enables async scheduling and does not enable multi-token prediction (MTP) speculative decoding.
The profile retains --gpu-memory-utilization 0.4, a 262144-token context window, 4 concurrent sequences, an 8192-token batch limit, chunked prefill, prefix caching, and its registered parsers and acceleration backends.
On DGX Station, accepting the installer express prompt selects NEMOCLAW_VLLM_MODEL=nemotron-3-ultra-550b-a55b.
Express uses the single-Station Ultra recipe unless a trusted reciprocal pair qualifies for distributed serving.
Refer to Set Up vLLM on Two DGX Stations for pair qualification, lifecycle receipts, runtime topology, and network controls.
On DGX Spark, Express selects the pinned two-node profile when exactly one pretrusted reciprocal pair qualifies and no inference intent or related runtime conflicts. Refer to Set Up vLLM on Two DGX Sparks for automatic and explicit selection, profile requirements, and cleanup behavior.
To select the existing deepseek-v4-flash recipe while retaining the same one-confirmation express flow, run:
The flag requires an interactive terminal; in a curl | bash pipeline, /dev/tty must be available.
Without terminal access, the installer stops before it installs Docker or build dependencies instead of silently continuing with another configuration.
The registered single-Station Ultra recipe tracks the official DGX Station deployment guide and configures the pinned model revision, CPU offload, 16 GB of shared memory, memory/stack ulimits, MTP speculative decoding, and the Nemotron reasoning and tool-call parsers.
Before managed vLLM setup on DGX Station, follow Prepare DGX Station to Install NemoClaw. For the current support status and direct GPU policy boundaries, see Platform Support.
Run Non-Interactive Onboarding
On DGX Spark, non-interactive onboarding can select local vLLM when NEMOCLAW_PROVIDER is unset and no provider is recorded for the sandbox.
It reuses a running local vLLM server first.
If no server is running, it selects the managed install or start entry.
If neither local entry is available, it selects NVIDIA Endpoints.
On DGX Station and other hosts, an unset provider keeps NVIDIA Endpoints as the automatic default.
Use an already-running server.
Install or start managed vLLM on a supported host.
On DGX Spark and DGX Station, NEMOCLAW_PROVIDER=install-vllm is sufficient for a non-interactive run.
Add NEMOCLAW_EXPERIMENTAL=1 on a generic Linux NVIDIA GPU host.
Non-interactive runs use the profile default unless you set NEMOCLAW_VLLM_MODEL.
The commands above invoke nemo-deepagents onboard directly, so a DGX Station run with no model or peer selects the deepseek-v4-flash profile default.
Supplying NEMOCLAW_PROVIDER=install-vllm to the shell installer enters the Station host-preparation boundary while retaining that profile default.
For paired non-interactive setup, follow Set Up vLLM on Two DGX Stations.
For a headless DGX Station setup that selects DeepSeek V4 Flash explicitly, use the environment-variable path instead of --station-deepseek.
For a headless or CI first install on a qualifying DGX Station after host preparation, set the same managed-vLLM recipe in the installer environment.
Select a Managed Model
Set NEMOCLAW_VLLM_MODEL=<slug> before onboarding to select a model without prompting.
NemoClaw applies the registered vllm serve arguments, including the reasoning parser, tool-call parser, and --max-model-len.
Slugs are case-insensitive, and NemoClaw also accepts the full Hugging Face model ID. An unrecognized value fails before image or model downloads and prints the valid slugs. A recognized override that does not support the detected host also fails before image or model downloads. The error names the selected model and detected host profile. A model whose quantized checkpoint needs a newer GPU than the host provides also fails before those downloads. Follow Check GPU Compute Capability for that check.
Gated models require a Hugging Face token and license acceptance. Follow Authenticate Hugging Face Downloads before onboarding so NemoClaw can forward the token temporarily to the one-shot model downloader.
NemoClaw also accepts HUGGING_FACE_HUB_TOKEN as a compatibility alias.
The host checks the token before docker pull, so a missing or empty token stops onboarding before bandwidth is spent on a request that would return 401.
Add Managed Serve Arguments
Set NEMOCLAW_VLLM_EXTRA_ARGS_JSON to a JSON array of individual non-blank vllm serve tokens for operator-owned options and overrides.
NemoClaw trims and validates the array before downloads, shell-quotes each token, and appends the tokens after the registry defaults.
If the selected image does not support an argument, the managed container exits and NemoClaw prints the log tail.
To bound resource use while investigating long-context workflows on a single DGX Spark, select the Qwen profile. The following override disables async scheduling and lowers the context window, concurrent-sequence limit, and batch limit.
This override reduces request concurrency and context capacity, so it can reduce throughput. It does not guarantee protection against a DGX Spark host freeze.
Related Topics
- Choose a Local Inference Server to compare vLLM with Ollama and NVIDIA NIM.
- Set Up vLLM on Two DGX Sparks for the Experimental distributed DeepSeek V4 Flash profile.
- Set Up vLLM on Two DGX Stations for the Deferred distributed-serving workflow.
- Configure Inference Timeouts when model startup or validation needs more time.
- Verify the Inference Route after setup.