InferenceCustom Endpoints

Set Up an OpenAI-Compatible Endpoint

View as Markdown

Use the custom OpenAI-compatible provider for servers that implement /v1/chat/completions or a compatible /v1/responses API. Examples include vLLM, TensorRT-LLM, llama.cpp, LocalAI, and other compatible servers.

The agent connects to inference.local inside the sandbox. OpenShell forwards that traffic to the endpoint configured during onboarding.

Start the Server

Start the compatible server before onboarding. The following example starts vLLM on port 8000.

$vllm serve meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 8000

Port 8000 is included in the local-inference network policy preset.

The examples on this page listen on every IPv4 interface so both host-side onboarding and a containerized gateway can reach the server. When the server does not require authentication, use a host firewall with default-deny inbound rules. Allow TCP port 8000 only from the OpenShell Docker subnet to its gateway address, keep loopback access, and deny the port on every other interface. Do not expose the port to your LAN or the internet.

Run Onboarding

Start the onboard wizard.

$nemohermes onboard

Select Other OpenAI-compatible endpoint. Enter the server base URL and the model ID reported by the server.

Use http://localhost:8000/v1 when the default Docker-driver topology makes the host endpoint reachable at that address. For a containerized OpenShell gateway that cannot reach host localhost, use the host-gateway URL for your environment, commonly http://host.openshell.internal:8000/v1.

For host-routable endpoints, NemoClaw verifies the API, tool-calling, and streaming paths before sandbox creation. The host-side endpoint probe is skipped for the sandbox-internal host.openshell.internal alias, so verify the runtime route after onboarding when you use that address.

The wizard prompts for an API key. Enter any non-empty placeholder such as dummy when the server does not require authentication.

Refer to Choose a Compatible Inference API for the probe order and runtime API selection.

Serve a Raw Model File

Start a compatible server for a raw model file instead of passing the file path to NemoClaw. The Ollama provider accepts Ollama model tags and does not accept a raw .gguf path.

The following example starts llama-server with a GGUF model.

$llama-server \
>  -m /models/NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf \
>  --host 0.0.0.0 \
>  --port 8000 \
>  -c 16384 \
>  -ngl 999 \
>  --parallel 1 \
>  --chat-template chatml

During onboarding, select Other OpenAI-compatible endpoint. Enter the server base URL and the model ID returned by /v1/models. Use the model ID, not the raw file path.

For the example above, the server commonly reports NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf as its model ID.

Run Non-Interactive Onboarding

Start the endpoint before running the non-interactive command because onboarding validates the server. Set NEMOCLAW_REASONING=true when the endpoint serves a reasoning-only model.

Reasoning mode validates only /v1/chat/completions and does not verify tool calling or streaming. Enable it only when the endpoint supports the capabilities your agent requires.

$NEMOCLAW_PROVIDER=custom \
>  NEMOCLAW_ENDPOINT_URL=http://localhost:8000/v1 \
>  NEMOCLAW_MODEL=meta-llama/Llama-3.1-8B-Instruct \
>  COMPATIBLE_API_KEY=dummy \
>  nemohermes onboard --non-interactive

For the raw model example, use the ID returned by /v1/models.

$NEMOCLAW_PROVIDER=custom \
>  NEMOCLAW_ENDPOINT_URL=http://localhost:8000/v1 \
>  NEMOCLAW_MODEL=NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf \
>  COMPATIBLE_API_KEY=dummy \
>  nemohermes onboard --non-interactive
VariablePurpose
NEMOCLAW_PROVIDERSet to custom.
NEMOCLAW_ENDPOINT_URLBase URL of the server.
NEMOCLAW_MODELModel ID reported by the server.
NEMOCLAW_REASONINGEnables reasoning-only validation with the case-insensitive true values true, 1, yes, and y.
COMPATIBLE_API_KEYEndpoint API key, or a non-empty placeholder when authentication is not required.