Inference

Switch Inference Models

View as Markdown

Change the active inference model for a NemoClaw-managed sandbox. OpenClaw and Hermes can update the running sandbox route without a restart. Deep Agents sandboxes should use the recreate path so the OpenShell route and /sandbox/.deepagents/config.toml stay aligned.

Choose one of these paths:

  • Runtime switch: use nemo-deepagents inference set for OpenClaw and Hermes when the target provider is already registered on the sandbox. This updates the OpenShell route and synchronizes the running agent config without recreating the sandbox.
  • Fresh recreation: use nemo-deepagents onboard --fresh --name <sandbox-name> --recreate-sandbox for a consistent all-agent path. This is required for Deep Agents provider or model changes, and it also applies to any agent when you change build-time settings.
  • Recorded rebuild: use nemo-deepagents <name> rebuild when you want to recreate the sandbox from the provider and model already recorded in the registry. That rebuild flow may use an internal resume handoff, but it does not choose a new provider or model.
  • Interrupted recovery: use --resume only to recover an interrupted onboarding session.

Before You Start

  • Start from an existing NemoClaw sandbox created with nemo-deepagents onboard.
  • Keep the OpenShell CLI on your PATH, which NemoClaw uses internally.
  • For runtime switches, select a default sandbox or pass --sandbox <name> to each nemo-deepagents inference set command when more than one sandbox is registered.
  • For fresh recreation, pass the existing sandbox name when you want to replace that sandbox.
  • For side-by-side validation, pass a new sandbox name and switch the default after the new sandbox passes validation.
  • Keep the required provider credential available in your host shell before rerunning onboarding.

If you run nemo-deepagents inference set before NemoClaw has a selected sandbox, the CLI exits with No sandbox selected. Run nemo-deepagents onboard first, then retry the switch.

OpenShell currently exposes one inference route per gateway, not one route per sandbox. All registered sandboxes on the same gateway must use the same provider and model. Custom compatible routes must also use the same endpoint and API family. An OpenClaw native-Anthropic route and a Hermes OpenAI-frontend route therefore cannot share a gateway, even when their custom endpoint and model are otherwise identical. NemoClaw checks stopped sandboxes because they depend on that route when restarted. If a registered same-gateway sandbox lacks durable provider or model metadata, or a custom route lacks durable endpoint or API-family metadata, NemoClaw fails closed until you remove and re-onboard that sandbox with complete route metadata. When a route conflicts, onboarding, runtime switching, and connect-time repair exit non-zero before changing the gateway and name the affected sandboxes. Align the routes, remove the conflicting sandbox, or onboard it with another NEMOCLAW_GATEWAY_PORT.

Switch to a Different Model

For Deep Agents, use a fresh named recreation when you need to switch providers or models. The managed dcode runtime reads /sandbox/.deepagents/config.toml, which NemoClaw writes during onboarding and rebuilds. The recreate path keeps that file, the OpenShell route, and the host registry in sync while runtime switching for Deep Agents is still being hardened. The --resume flag is not a provider or model switch path for any variant. After completed onboarding, sessions are no longer resumable. During an interrupted resume, NemoClaw reuses the recorded provider and model. Pass the existing sandbox name when you want to replace that sandbox.

Deep Agents Recreate Path

Use the fresh recreate path for Deep Agents provider or model changes:

$nemo-deepagents onboard --fresh --name <sandbox-name> --recreate-sandbox

The wizard prompts for provider and model choices, backs up the existing sandbox state, recreates the named sandbox, and rewrites the managed dcode config for the new route. If you want to keep the current sandbox until the new route passes validation, create a side-by-side sandbox first:

$nemo-deepagents onboard --fresh --name <new-sandbox-name>
$nemo-deepagents use <new-sandbox-name>

API Family Sync

For Deep Agents, NemoClaw keeps the managed runtime on Chat Completions with use_responses_api = false. NEMOCLAW_PREFERRED_API=openai-responses can affect compatible-endpoint onboarding probes for other variants, but it does not switch managed dcode to the Responses API.

Cross-Provider Switching

Cross-provider changes for Deep Agents should use sandbox recreation instead of nemo-deepagents inference set. The recreate flow validates the target provider, rewrites /sandbox/.deepagents/config.toml, preserves manifest-defined state, and rebuilds managed MCP projection state from the host registry.

$nemo-deepagents onboard --fresh --name <sandbox-name> --recreate-sandbox

Verify the Active Model

Use nemo-deepagents inference get to print the provider and model the gateway is currently routing to. Run it before nemo-deepagents inference set to confirm the starting state, or after a switch to verify the new route.

$nemo-deepagents inference get

Expected output:

Provider: nvidia-prod
Model:    nvidia/nemotron-3-super-120b-a12b

Pass --json for machine-readable output.

$nemo-deepagents inference get --json

Expected output:

1{
2  "provider": "nvidia-prod",
3  "model": "nvidia/nemotron-3-super-120b-a12b"
4}

The command exits non-zero with OpenShell inference route is not configured. when the gateway has no registered inference route. Run nemo-deepagents onboard to configure one.

Run the status command when you also need sandbox, service, and messaging health:

$nemo-deepagents <name> status

The status output includes the active provider, model, and endpoint with the rest of the sandbox state.

Notes

  • The host keeps provider credentials.
  • The sandbox continues to use inference.local.
  • Use nemo-deepagents onboard --fresh --name <sandbox-name> --recreate-sandbox for provider, model, context window, max tokens, reasoning mode, timeout, or image-content changes.
  • The managed dcode runtime uses Chat Completions through /sandbox/.deepagents/config.toml with use_responses_api = false.
  • Local Ollama and local vLLM routes use local provider tokens rather than OPENAI_API_KEY.