> This page is for version 0.11.
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt
> - 0.11: https://docs.nvidia.com/nemo-helix/v0.11/llms.txt
> - 0.10: https://docs.nvidia.com/nemo-helix/v0.10/llms.txt
> - 0.9: https://docs.nvidia.com/nemo-helix/v0.9/llms.txt
> - 0.8: https://docs.nvidia.com/nemo-helix/v0.8/llms.txt
> - 0.7: https://docs.nvidia.com/nemo-helix/v0.7/llms.txt
> - 0.6: https://docs.nvidia.com/nemo-helix/v0.6/llms.txt
> - 0.5: https://docs.nvidia.com/nemo-helix/v0.5/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.

# Deploy Models

Deploy models from NGC or HuggingFace. Register external providers like OpenAI or NVIDIA Build.

Model deployments on the control plane use the deployments plugin (Docker, Kubernetes, or OpenShell executors — no subprocess backend). The default local config uses the Docker executor, so local model serving needs a running Docker daemon. See [Local Docker and Executors](/nemo-helix/v0.11/documentation/get-started/local-docker-and-executors).

> **Note**
>
> Resource names for deployments, deployment configs, and providers must contain only letters (a-z, A-Z), digits (0-9), underscores, hyphens, and dots. For example: `llama-3.1-8b`, `my-custom-model`, `qwen-fs-config`.

## Choose an Inference Engine

The `engine` field on a deployment config selects how the platform serves the model. We recommend vLLM unless you need a NIM.

| Engine and Image                           | Platform Default Image                                                                                                     | Use When                                                                    | Supported Models                                                                                                                          |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| **vLLM** (`engine: vllm`), recommended     | `docker.io/vllm/vllm-openai`, pinned to a tested tag                                                                       | Hugging Face checkpoints and models trained with Customizer                 | Architectures the bundled vLLM version supports; see [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html) |
| **Model-free NIM** (`engine: nim`)         | `nvcr.io/nim/nvidia/vllm-model-free-nim:latest`; set `image_name` to `nvcr.io/nim/nvidia/sglang-model-free-nim` for SGLang | You want a NIM for a Hugging Face checkpoint that has no model-specific NIM | [Model-free NIM support matrix](https://docs.nvidia.com/nim/large-language-models/latest/reference/support-matrix.html#model-free-nim)    |
| **Model-specific NIM** (`engine: nim`)     | None; set `image_name` and `image_tag`                                                                                     | Production serving of a model that NVIDIA publishes as a NIM                | [NIM support matrix](https://docs.nvidia.com/nim/large-language-models/latest/reference/support-matrix.html)                              |
| **Multi-LLM NIM** (`engine: nim`), NIM 1.x | None; set `image_name` to `nvcr.io/nim/nvidia/llm-nim`                                                                     | Existing NIM 1.x deployments only                                           | [Multi-LLM NIM supported architectures](https://docs.nvidia.com/nim/large-language-models/1.15.0/supported-architectures.html)            |

The model-free NIM is the NIM 2.x successor to the multi-LLM NIM. It is not a rename: each model-free NIM image bundles one engine, vLLM (`vllm-model-free-nim`) or SGLang (`sglang-model-free-nim`), and reads a different configuration than NIM 1.x. Use the model-free NIM instead of the multi-LLM NIM for new deployments; see the [NIM 1.x migration guide](https://docs.nvidia.com/nim/large-language-models/latest/reference/1.x-migration-guide.html). The platform sets the configuration that each NIM version reads.

> **Warning**
>
> The multi-LLM NIM cannot serve merged LoRA checkpoints from Customizer (`finetuning_type: lora_merged`). The merge step saves the model config and tokenizer with transformers 5, and the multi-LLM NIM 1.15 uses transformers 4.56. Deploy those models with [vLLM](#deploy-with-vllm) or the model-free NIM.

The CLI and Python SDK connect to the deployment in your current CLI context. To connect to a local or remote deployment, see [Connect with CLI Contexts](/nemo-helix/v0.11/documentation/reference/cli-reference/connect-to-deployments).

#### CLI

```bash
# Confirm the deployment and workspace in the current context
nemo config view
```

#### Python SDK

```python
from nemo_helix_plugin.client.client import NemoClient

client = NemoClient.from_config()
```

> **Note**
>
> These examples use the `default` workspace. If your current CLI context uses a different workspace, run `nemo config set --workspace default` first. To check, run `nemo config view`.

The Python SDK examples on this page also use these imports:

```python
import os
from pathlib import Path

from pydantic import SecretStr

from nemo_helix_plugin.files.metadata import FilesetMetadata, ModelMetadataContent, ToolCallingMetadataContent
from nemo_helix_plugin.files.storage_config import HuggingfaceStorageConfig
from nemo_helix_plugin.files.types import CreateFilesetRequest, UpdateFilesetRequest
from nemo_helix_plugin.inference_gateway.client import InferenceGatewayClient
from nemo_helix_plugin.inference_gateway.types import JsonBody
from nemo_helix_plugin.models.types import (
    ContainerExecutorConfig,
    CreateModelAdapterRequest,
    CreateModelDeploymentConfigRequest,
    CreateModelDeploymentRequest,
    CreateModelEntityRequest,
    CreateModelProviderRequest,
    Engine,
    FinetuningType,
    ModelDeploymentConfigModelSpec,
    ToolCallConfig,
)
from nemo_helix_plugin.schema import SecretRef
from nemo_helix_plugin.secrets.types import HelixSecretCreateRequest
```

---

## Add External Providers

Register external inference APIs like NVIDIA Build or OpenAI.

### NVIDIA Build

By default, the platform pre-configures an external provider for NVIDIA Build named `nvidia-build` in the `system` workspace.
The example below demonstrates how to recreate it in your own workspace.
For disambiguation purposes, this example names the manually-created version `my-nvidia-build`.

#### CLI

```bash
# Store API key
echo "$NVIDIA_API_KEY" | nemo secrets create "nvidia-api-key" --from-file -

# Create provider
nemo inference providers create "my-nvidia-build" \
  --host-url "${NVIDIA_BASE_URL:-https://integrate.api.nvidia.com/v1}" \
  --api-key-secret-name "nvidia-api-key"

nemo wait inference provider my-nvidia-build

# Test using interactive chat
nemo chat -m nvidia/llama-3.3-nemotron-super-49b-v1 'Hello!' \
  --provider my-nvidia-build
```

#### Python SDK

```python
import os

# Store API key
client.secrets.create_secret(
    body=HelixSecretCreateRequest(
        name="nvidia-api-key",
        value=SecretStr(os.environ["NVIDIA_API_KEY"]),
    ),
)

# Create provider
provider = client.models.create_provider(
    body=CreateModelProviderRequest(
        name="my-nvidia-build",
        host_url=os.environ.get("NVIDIA_BASE_URL", "https://integrate.api.nvidia.com/v1").rstrip("/"),
        api_key_secret_name="nvidia-api-key",
    ),
).data()

client.models.wait_for_provider_status(provider_name="my-nvidia-build")

# Use provider routing
response = InferenceGatewayClient.from_client(client).provider_post(
    name="my-nvidia-build",
    trailing_uri="v1/chat/completions",
    body=JsonBody({
        "model": "nvidia/nemotron-3.5-lightning-30b-a3b",
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 100,
    }),
)
```

### OpenAI

#### CLI

```bash
# Store API key
echo "$OPENAI_API_KEY" | nemo secrets create "openai-api-key" --from-file -

# Create provider with enabled models
nemo inference providers create "openai" \
  --host-url "https://api.openai.com/v1" \
  --api-key-secret-name "openai-api-key" \
  --enabled-models "gpt-4" \
  --enabled-models "gpt-3.5-turbo"

nemo wait inference provider openai

# Test using interactive chat
nemo chat -m gpt-4 'Hello!' \
  --provider openai
```

#### Python SDK

```python
client.secrets.create_secret(
    body=HelixSecretCreateRequest(
        name="openai-api-key",
        value=SecretStr(os.environ["OPENAI_API_KEY"]),
    ),
)

provider = client.models.create_provider(
    body=CreateModelProviderRequest(
        name="openai",
        host_url="https://api.openai.com/v1",
        api_key_secret_name="openai-api-key",
        enabled_models=["gpt-4", "gpt-3.5-turbo"],
    ),
).data()

client.models.wait_for_provider_status(provider_name="openai")

# Use provider routing
response = InferenceGatewayClient.from_client(client).provider_post(
    name="openai",
    trailing_uri="v1/chat/completions",
    body=JsonBody({
        "model": "gpt-4",
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 100,
    }),
)
```

### Anthropic

Anthropic's `/v1/messages` API expects the API key in an `X-Api-Key:` header (not `Authorization: Bearer`) and requires an `anthropic-version` header on every request. Use `--auth-header-format` (a literal header format that must contain exactly one `<auth_secret>` placeholder) to override the default `Authorization: Bearer <auth_secret>` and pass the API-version pin via `--default-extra-headers`. Without these, Anthropic rejects every request with 401.

#### CLI

```bash
# Store API key
echo "$ANTHROPIC_API_KEY" | nemo secrets create "anthropic-api-key" --from-file -

# Create provider — override the default Bearer header and pin the API version
nemo inference providers create "anthropic" \
  --host-url "https://api.anthropic.com" \
  --api-key-secret-name "anthropic-api-key" \
  --auth-header-format "X-Api-Key: <auth_secret>" \
  --default-extra-headers '{"anthropic-version": "2023-06-01"}'

nemo wait inference provider anthropic
```

#### Python SDK

```python
# Store API key
client.secrets.create_secret(
    body=HelixSecretCreateRequest(
        name="anthropic-api-key",
        value=SecretStr(os.environ["ANTHROPIC_API_KEY"]),
    ),
)

# Create provider — override the default Bearer header and pin the API version
provider = client.models.create_provider(
    body=CreateModelProviderRequest(
        name="anthropic",
        host_url="https://api.anthropic.com",
        api_key_secret_name="anthropic-api-key",
        auth_header_format="X-Api-Key: <auth_secret>",
        default_extra_headers={"anthropic-version": "2023-06-01"},
    ),
).data()

client.models.wait_for_provider_status(provider_name="anthropic")
```

`<auth_secret>` is substituted with the resolved secret value at request time. The legacy `{{ auth_secret }}` spelling is still accepted.

---

## Deploy from NGC

Deploy pre-built NIM containers from NGC.

### Deploy Llama 3.2 1B

#### CLI

```bash
nemo inference deployment-configs create "llama-3-2-1b-config" \
  --engine "nim" \
  --model-spec '{
    "model_name": "meta/llama-3.2-1b-instruct"
}' \
  --executor-config '{
    "gpu": 1,
    "image_name": "nvcr.io/nim/meta/llama-3.2-1b-instruct",
    "image_tag": "1.8.6"
}'

nemo inference deployments create "llama-3-2-1b-deployment" \
  --config "llama-3-2-1b-config"

nemo wait inference deployment llama-3-2-1b-deployment

nemo chat -m meta/llama-3.2-1b-instruct 'Hello!' \
  --provider llama-3-2-1b-deployment \
  --max-tokens 100
```

#### Python SDK

```python
config = client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name="llama-3-2-1b-config",
        engine=Engine.NIM,
        model_spec=ModelDeploymentConfigModelSpec(model_name="meta/llama-3.2-1b-instruct"),
        executor_config=ContainerExecutorConfig(
            gpu=1,
            image_name="nvcr.io/nim/meta/llama-3.2-1b-instruct",
            image_tag="1.8.6",
        ),
    ),
).data()

deployment = client.models.create_deployment(
    body=CreateModelDeploymentRequest(
        name="llama-3-2-1b-deployment",
        config="llama-3-2-1b-config",
    ),
).data()

client.models.wait_for_deployment_status(
    deployment_name="llama-3-2-1b-deployment",
    desired_status="READY",
)

response = InferenceGatewayClient.from_client(client).provider_post(
    name="llama-3-2-1b-deployment",
    trailing_uri="v1/chat/completions",
    body=JsonBody({
        "model": "meta/llama-3.2-1b-instruct",
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 100,
    }),
)
```

### Deploy NemoGuard JailbreakDetect

Deploy classification NIMs like NemoGuard for content safety. Uses the `/v1/classify` endpoint instead of chat completions.

#### CLI

```bash
nemo inference deployment-configs create "nemoguard-jailbreak-config" \
  --engine "nim" \
  --model-spec '{}' \
  --executor-config '{
    "gpu": 1,
    "image_name": "nvcr.io/nim/nvidia/nemoguard-jailbreak-detect",
    "image_tag": "1.10.1"
}'

nemo inference deployments create "nemoguard-jailbreak-deployment" \
  --config "nemoguard-jailbreak-config"

nemo wait inference deployment nemoguard-jailbreak-deployment

nemo inference gateway provider post v1/classify "nemoguard-jailbreak-deployment" \
  --body '{"input": "Tell me about vacation spots in Hawaii."}'
```

#### Python SDK

```python
config = client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name="nemoguard-jailbreak-config",
        engine=Engine.NIM,
        model_spec=ModelDeploymentConfigModelSpec(),
        executor_config=ContainerExecutorConfig(
            gpu=1,
            image_name="nvcr.io/nim/nvidia/nemoguard-jailbreak-detect",
            image_tag="1.10.1",
        ),
    ),
).data()

deployment = client.models.create_deployment(
    body=CreateModelDeploymentRequest(
        name="nemoguard-jailbreak-deployment",
        config="nemoguard-jailbreak-config",
    ),
).data()

client.models.wait_for_deployment_status(
    deployment_name="nemoguard-jailbreak-deployment",
    desired_status="READY",
)

response = InferenceGatewayClient.from_client(client).provider_post(
    name="nemoguard-jailbreak-deployment",
    trailing_uri="v1/classify",
    body=JsonBody({"input": "Tell me about vacation spots in Hawaii."}),
)
```

---

## Deploy from HuggingFace

> **Warning**
>
> A NIM deployment config that sets no image uses the platform default, the [model-free NIM](https://docs.nvidia.com/nim/large-language-models/latest/deployment/model-free-nim.html) (`nvcr.io/nim/nvidia/vllm-model-free-nim:latest`), which only serves architectures that its vLLM version supports. Check its [support matrix](https://docs.nvidia.com/nim/large-language-models/latest/reference/support-matrix.html#model-free-nim) before deploying. If your model architecture is not supported, use a model-specific NIM image (see [Deploy from NGC](#deploy-from-ngc)) or [deploy with vLLM](#deploy-with-vllm).
>
> The model-free NIM is the NIM 2.x successor to the [multi-LLM NIM](https://docs.nvidia.com/nim/large-language-models/1.15.0/introduction.html) (`nvcr.io/nim/nvidia/llm-nim`); see [Choose an Inference Engine](#choose-an-inference-engine). To use the multi-LLM NIM, set `image_name` to `nvcr.io/nim/nvidia/llm-nim` and check its [supported architectures](https://docs.nvidia.com/nim/large-language-models/1.15.0/supported-architectures.html). It uses the `latest` tag unless you set `image_tag`, and it cannot serve merged LoRA checkpoints from Customizer.
>
> To serve models trained with Customizer, we recommend [deploying with vLLM](#deploy-with-vllm).

> **Note**
>
> On Kubernetes, the platform downloads the model's files into a volume of `executor_config.disk_size`, which defaults to `50Gi`. That fits most models. For a larger checkpoint, set `disk_size` above the size of the model's files on Hugging Face; otherwise the download runs out of space and the deployment never becomes ready. For example, Nemotron 3.5 Super VL BF16 (249 GB) needs at least `300Gi`.

You can register a HuggingFace model through the Files service. This creates a fileset that acts as a proxy. The Files service handles authentication and caches the weights on first download, so subsequent deployments start faster.

#### CLI

```bash
# (Optional) Create a HuggingFace token secret for private models.
# Public models like Qwen do not require a token.
echo "$HF_TOKEN" | nemo secrets create "hf-token-secret" --from-file -

# Create a fileset pointing to the HuggingFace model.
# "token_secret" is optional — only needed for private/gated models.
nemo files filesets create "qwen-2-5-1-5b" \
  --storage '{
"type": "huggingface",
"repo_id": "Qwen/Qwen2.5-1.5B-Instruct",
"repo_type": "model",
"token_secret": "hf-token-secret"
}'

# Register a model entity referencing the fileset
nemo models create "qwen-2-5-1-5b" \
  --fileset "default/qwen-2-5-1-5b"

# Create deployment config pointing to the model entity
nemo inference deployment-configs create "qwen-fs-config" \
  --engine "nim" \
  --model-spec '{
    "model_namespace": "default",
    "model_name": "qwen-2-5-1-5b"
}' \
  --executor-config '{
    "gpu": 1
}'

nemo inference deployments create "qwen-fs-deployment" \
  --config "qwen-fs-config"

nemo wait inference deployment qwen-fs-deployment

nemo chat -m default/qwen-2-5-1-5b 'Hello!' \
  --provider qwen-fs-deployment \
  --max-tokens 100
```

#### Python SDK

```python
# (Optional) Create a HuggingFace token secret for private models.
# Public models like Qwen do not require a token.
client.secrets.create_secret(
    body=HelixSecretCreateRequest(
        name="hf-token-secret",
        value=SecretStr(os.environ["HF_TOKEN"]),
    ),
)

# Create a fileset pointing to the HuggingFace model.
# "token_secret" is optional — only needed for private/gated models.
client.files.create_fileset(
    body=CreateFilesetRequest(
        name="qwen-2-5-1-5b",
        storage=HuggingfaceStorageConfig(
            repo_id="Qwen/Qwen2.5-1.5B-Instruct",
            repo_type="model",
            token_secret=SecretRef("hf-token-secret"),
        ),
    ),
    exist_ok=True,
)

# Register a model entity referencing the fileset
client.models.create_model(
    body=CreateModelEntityRequest(name="qwen-2-5-1-5b", fileset="default/qwen-2-5-1-5b"),
)

# Create deployment config pointing to the model entity
config = client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name="qwen-fs-config",
        engine=Engine.NIM,
        model_spec=ModelDeploymentConfigModelSpec(
            model_namespace="default",
            model_name="qwen-2-5-1-5b",
        ),
        executor_config=ContainerExecutorConfig(gpu=1),
    ),
).data()

# Deploy — no hf_token_secret_name needed
deployment = client.models.create_deployment(
    body=CreateModelDeploymentRequest(name="qwen-fs-deployment", config="qwen-fs-config"),
).data()

client.models.wait_for_deployment_status(
    deployment_name="qwen-fs-deployment",
    desired_status="READY",
)

response = InferenceGatewayClient.from_client(client).provider_post(
    name="qwen-fs-deployment",
    trailing_uri="v1/chat/completions",
    body=JsonBody({
        "model": "default/qwen-2-5-1-5b",
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 100,
    }),
)
```

> **Tip**
>
> The `fileset` format is `<workspace>/<fileset-name>`. This tells the deployment system to pull weights from the Files service, which proxies the download from HuggingFace using the fileset `token_secret`. For public models like Qwen, the `token_secret` on the fileset is optional.

---

## Deploy with vLLM

Serve any HuggingFace-format model with [vLLM](https://docs.vllm.ai/) by setting `engine: "vllm"` on the deployment config. The model is registered through the Files service exactly as in [Deploy from HuggingFace](#deploy-from-huggingface) — only the `engine` field on the deployment config changes. Tensor parallelism is computed automatically from the GPU count and the model's architecture.

This example deploys [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B) with vLLM.

#### CLI

```bash
# Create a fileset pointing to the HuggingFace model.
# Qwen models are public — token_secret is optional.
nemo files filesets create "qwen3-1-7b" \
  --storage '{
    "type": "huggingface",
    "repo_id": "Qwen/Qwen3-1.7B",
    "repo_type": "model"
  }'

# Register a model entity referencing the fileset
nemo models create "qwen3-1-7b" \
  --fileset "default/qwen3-1-7b"

# Create deployment config with engine "vllm"
nemo inference deployment-configs create "qwen3-vllm-config" \
  --engine "vllm" \
  --model-spec '{
    "model_namespace": "default",
    "model_name": "qwen3-1-7b"
  }' \
  --executor-config '{
    "gpu": 1
  }'

nemo inference deployments create "qwen3-vllm-deployment" \
  --config "qwen3-vllm-config"

nemo wait inference deployment qwen3-vllm-deployment

nemo chat -m default/qwen3-1-7b 'Hello!' \
  --provider qwen3-vllm-deployment \
  --max-tokens 100
```

#### Python SDK

```python
# Create a fileset pointing to the HuggingFace model.
# Qwen models are public — token_secret is optional.
client.files.create_fileset(
    body=CreateFilesetRequest(
        name="qwen3-1-7b",
        storage=HuggingfaceStorageConfig(repo_id="Qwen/Qwen3-1.7B", repo_type="model"),
    ),
    exist_ok=True,
)

# Register a model entity referencing the fileset
client.models.create_model(
    body=CreateModelEntityRequest(name="qwen3-1-7b", fileset="default/qwen3-1-7b"),
)

# Create deployment config with engine "vllm"
config = client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name="qwen3-vllm-config",
        engine=Engine.VLLM,
        model_spec=ModelDeploymentConfigModelSpec(
            model_namespace="default",
            model_name="qwen3-1-7b",
        ),
        executor_config=ContainerExecutorConfig(gpu=1),
    ),
).data()

deployment = client.models.create_deployment(
    body=CreateModelDeploymentRequest(
        name="qwen3-vllm-deployment",
        config="qwen3-vllm-config",
    ),
).data()

client.models.wait_for_deployment_status(
    deployment_name="qwen3-vllm-deployment",
    desired_status="READY",
)

response = InferenceGatewayClient.from_client(client).provider_post(
    name="qwen3-vllm-deployment",
    trailing_uri="v1/chat/completions",
    body=JsonBody({
        "model": "default/qwen3-1-7b",
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 100,
    }),
)
```

### Deploy with LoRA Adapters

vLLM deployments support LoRA adapters with hot-reload: enable LoRA on the deployment config, then register one or more adapters against the base model entity. An adapter sidecar delivers each enabled adapter to the running server, and vLLM serves it on a per-request basis. Adapters are registered on the **base model entity**, not as separate models.

A deployment loads adapters up to a maximum rank and refuses adapters of a higher rank. Set the maximum to at least the highest rank you serve:

| Engine and Image       | Default Maximum Rank                                                                                                                                      | To Raise It                                                                             |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| vLLM                   | 32, set by the platform (vLLM's own default is 16). [Nemotron 3.5 Super VL](#deploy-nemotron-super-vl) recipe adapters are rank 64, so set `64` for them. | `executor_config.additional_args`: `["--max-lora-rank", "<rank>"]`                      |
| Model-free NIM, vLLM   | 16                                                                                                                                                        | `executor_config.additional_envs`: `{"NIM_PASSTHROUGH_ARGS": "--max-lora-rank <rank>"}` |
| Model-free NIM, SGLang | 16 if the server starts with no adapters; otherwise the highest rank among the adapters present when it starts                                            | `executor_config.additional_envs`: `{"NIM_PASSTHROUGH_ARGS": "--max-lora-rank <rank>"}` |
| Multi-LLM NIM          | 32                                                                                                                                                        | `executor_config.additional_envs`: `{"NIM_MAX_LORA_RANK": "<rank>"}`                    |

vLLM accepts `--max-lora-rank` values of 1, 8, 16, 32, 64, 128, 256, 320, and 512.

Each adapter is served under the name `<workspace>--<adapter-name>`. On the SGLang model-free NIM, the served name is `<base-model>:<workspace>--<adapter-name>`, for example `default/qwen3-1-7b:default--linear-algebra`, because SGLang answers a bare adapter name with the base model. The model provider's `served_models` lists the served name of each adapter.

This example serves the [linear-algebra LoRA](https://huggingface.co/premjatin/qwen-linear-algebra-coder) on top of Qwen3-1.7B.

#### CLI

```bash
# Base model fileset + entity (skip if already created above)
nemo files filesets create "qwen3-1-7b" \
  --storage '{
    "type": "huggingface",
    "repo_id": "Qwen/Qwen3-1.7B",
    "repo_type": "model"
  }'

nemo models create "qwen3-1-7b" \
  --fileset "default/qwen3-1-7b"

# Fileset for the LoRA adapter weights
nemo files filesets create "linear-algebra-lora" \
  --storage '{
    "type": "huggingface",
    "repo_id": "premjatin/qwen-linear-algebra-coder",
    "repo_type": "model"
  }'

# Deployment config with LoRA enabled.
# --max-lora-rank is only needed above the default of 32.
# Set it to the highest adapter rank you intend to serve.
nemo inference deployment-configs create "qwen3-lora-config" \
  --engine "vllm" \
  --model-spec '{
    "model_namespace": "default",
    "model_name": "qwen3-1-7b",
    "lora_enabled": true
  }' \
  --executor-config '{
    "gpu": 1,
    "additional_args": ["--max-lora-rank", "64"]
  }'

nemo inference deployments create "qwen3-lora-deployment" \
  --config "qwen3-lora-config"

nemo wait inference deployment qwen3-lora-deployment

# Register the adapter against the base model entity.
# The adapter is served under the name "<workspace>--<adapter-name>".
nemo models adapters create "qwen3-1-7b" "linear-algebra" \
  --fileset "default/linear-algebra-lora" \
  --finetuning-type lora \
  --enabled

# Target the adapter by its served name in the request body
nemo inference gateway provider post v1/chat/completions qwen3-lora-deployment \
  --body '{
    "model": "default--linear-algebra",
    "messages": [{"role": "user", "content": "Find the determinant of [[2, 1], [1, 3]]."}],
    "max_tokens": 256
  }'
```

#### Python SDK

```python
# Base model fileset + entity (skip if already created above)
client.files.create_fileset(
    body=CreateFilesetRequest(
        name="qwen3-1-7b",
        storage=HuggingfaceStorageConfig(repo_id="Qwen/Qwen3-1.7B", repo_type="model"),
    ),
    exist_ok=True,
)

client.models.create_model(
    body=CreateModelEntityRequest(name="qwen3-1-7b", fileset="default/qwen3-1-7b"),
)

# Fileset for the LoRA adapter weights
client.files.create_fileset(
    body=CreateFilesetRequest(
        name="linear-algebra-lora",
        storage=HuggingfaceStorageConfig(
            repo_id="premjatin/qwen-linear-algebra-coder",
            repo_type="model",
        ),
    ),
    exist_ok=True,
)

# Deployment config with LoRA enabled.
# --max-lora-rank is only needed above the default of 32.
# Set it to the highest adapter rank you intend to serve.
config = client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name="qwen3-lora-config",
        engine=Engine.VLLM,
        model_spec=ModelDeploymentConfigModelSpec(
            model_namespace="default",
            model_name="qwen3-1-7b",
            lora_enabled=True,
        ),
        executor_config=ContainerExecutorConfig(gpu=1, additional_args=["--max-lora-rank", "64"]),
    ),
).data()

deployment = client.models.create_deployment(
    body=CreateModelDeploymentRequest(
        name="qwen3-lora-deployment",
        config="qwen3-lora-config",
    ),
).data()

client.models.wait_for_deployment_status(
    deployment_name="qwen3-lora-deployment",
    desired_status="READY",
)

# Register the adapter against the base model entity.
# The adapter is served under the name "<workspace>--<adapter-name>".
client.models.create_model_adapter(
    model_name="qwen3-1-7b",
    body=CreateModelAdapterRequest(
        name="linear-algebra",
        fileset="default/linear-algebra-lora",
        finetuning_type=FinetuningType.LORA,
        enabled=True,
    ),
)

# Target the adapter by its served name in the request body
response = InferenceGatewayClient.from_client(client).provider_post(
    name="qwen3-lora-deployment",
    trailing_uri="v1/chat/completions",
    body=JsonBody({
        "model": "default--linear-algebra",
        "messages": [
            {"role": "user", "content": "Find the determinant of [[2, 1], [1, 3]]."}
        ],
        "max_tokens": 256,
    }),
)
```

> **Tip**
>
> The default vLLM image is configurable by the platform operator. To override the image per deployment, set `image_name` / `image_tag` on `executor_config`. To pass raw `vllm serve` flags (for example `--max-model-len`), use `additional_args` on `executor_config`.

### Deploy Nemotron 3.5 Super VL

Nemotron 3.5 Super VL is a vision-language model: 120B parameters with 12B active per token, on a hybrid Mamba and Transformer mixture-of-experts backbone. vLLM needs several model-specific flags to serve it. The platform does not add model-specific flags; set them in `executor_config.additional_args`. The flags below come from NVIDIA's [vLLM cookbook](https://github.com/NVIDIA-NeMo/Nemotron/blob/main/usage-cookbook/Nemotron-3.5-Super-VL/vllm_cookbook.ipynb) for the model, the tested serving recipe from the model team.

Register the model entity with `trust_remote_code` enabled. The checkpoint ships its own configuration code; without `trust_remote_code` vLLM cannot load the checkpoint and the platform cannot read its configuration to size tensor parallelism.

Two checkpoints are published:

| Checkpoint                                                                                                                          | Size   | Use It For                                                                                                                                                                 | GPUs             |
| ----------------------------------------------------------------------------------------------------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| [`nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16)   | 249 GB | The base model, LoRA adapters, and models fine-tuned with Customizer. Training uses the BF16 weights, so adapters and fine-tuned models must be served on this checkpoint. | 4x 80 GB or more |
| [`nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4) | 75 GB  | The base model only                                                                                                                                                        | 2x 80 GB or more |

For a released base model without LoRA, set:

```json
"additional_args": [
  "--max-model-len", "114688",
  "--reasoning-parser", "nemotron_v3",
  "--tool-call-parser", "qwen3_coder",
  "--enable-auto-tool-choice",
  "--mamba-ssm-cache-dtype", "float32",
  "--limit-mm-per-prompt", "{\"image\": 1, \"video\": 1}",
  "--speculative-config", "{\"method\": \"mtp\", \"num_speculative_tokens\": 3}"
]
```

With LoRA enabled (`model_spec.lora_enabled: true`), drop `--speculative-config` and raise the adapter limits:

```json
"additional_args": [
  "--max-model-len", "114688",
  "--reasoning-parser", "nemotron_v3",
  "--tool-call-parser", "qwen3_coder",
  "--enable-auto-tool-choice",
  "--mamba-ssm-cache-dtype", "float32",
  "--limit-mm-per-prompt", "{\"image\": 1, \"video\": 1}",
  "--max-lora-rank", "64",
  "--max-loras", "2"
]
```

For a fine-tuned checkpoint without LoRA, use the base model list without `--speculative-config` unless the checkpoint kept the MTP layer (see below).

| Flag                             | Value                                            | Why                                                                                                                                                                                                                                                                                                                                   |
| -------------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--max-model-len`                | `114688`                                         | Without it, vLLM sizes the KV cache for the model's native 1M-token context, cannot fit it, and exits at startup.                                                                                                                                                                                                                     |
| `--reasoning-parser`             | `nemotron_v3`                                    | Returns the model's thinking in the response's `reasoning` field, separate from the answer in `content`.                                                                                                                                                                                                                              |
| `--tool-call-parser`             | `qwen3_coder`                                    | Turns the model's tool calls into OpenAI `tool_calls`.                                                                                                                                                                                                                                                                                |
| `--enable-auto-tool-choice`      | (no value)                                       | Lets the model decide when to call a tool.                                                                                                                                                                                                                                                                                            |
| `--mamba-ssm-cache-dtype`        | `float32`                                        | Precision of the state the Mamba layers carry between tokens.                                                                                                                                                                                                                                                                         |
| `--limit-mm-per-prompt`          | `{"image": 1, "video": 1}`                       | At most one image and one video per request. Raise it to accept more, for example `{"image": 8, "video": 1}` for up to 8 page images.                                                                                                                                                                                                 |
| `--speculative-config`           | `{"method": "mtp", "num_speculative_tokens": 3}` | Faster generation with multi-token prediction; see below. Base model with an MTP layer, without LoRA.                                                                                                                                                                                                                                 |
| `--max-lora-rank`, `--max-loras` | `64`, `2`                                        | Recipe adapters are rank 64, above the platform default of 32, and vLLM otherwise loads one adapter per batch. Two slots let one batch use two adapters; vLLM swaps any other adapter in when a request names it. vLLM reserves GPU memory for every expert in each slot at startup, so a higher `--max-loras` needs more GPU memory. |

The speculative config turns on multi-token prediction (MTP). The released checkpoint includes an extra layer trained to predict the next few tokens. At each step vLLM uses that layer to guess the next 3 tokens, then runs the full model once to check all of them, and keeps every guess that matches what the full model would have generated. Responses are the same as without MTP; they arrive faster because one step can produce several tokens.

Add `--speculative-config` only for a checkpoint that has the MTP layer, with LoRA disabled:

* Not every checkpoint with this architecture has the layer. Nemotron 3.5 Bolt shares the architecture but has no MTP layer, so do not add `--speculative-config` for it.
* A fine-tuned checkpoint may not have the MTP layer. Automodel's Nemotron 3.5 Super VL recipes set `num_nextn_predict_layers: 0`, which builds the model without that layer, so a checkpoint saved from that training contains no MTP weights. If your fine-tuned checkpoint kept the MTP layer, you can add `--speculative-config`.
* With LoRA, the MTP layer comes from the base model and does not include the adapter's changes. NVIDIA's finance cookbook serves its adapters without MTP.

These flags are off by default. Add one to `additional_args` when its tradeoff suits your deployment:

| Flag                                   | What It Does                                                                        | Use It When                                                                                               | Tradeoff                                                                                                                                                                   |
| -------------------------------------- | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--enable-expert-parallel`             | Places whole experts on each GPU instead of splitting every expert across all GPUs. | You serve many concurrent requests. NVIDIA's cookbook uses it in its text-only throughput configurations. | The cookbook's image and LoRA configurations do not use it.                                                                                                                |
| `--skip-mm-profiling`                  | Skips the startup run that measures the vision encoder's peak memory.               | Startup time matters, or the profiling run fails at startup.                                              | vLLM no longer reserves memory for the encoder, so large images under load can run out of GPU memory. Leave headroom, for example with a lower `--gpu-memory-utilization`. |
| `--mm-encoder-attn-backend TORCH_SDPA` | Runs the vision encoder's attention with PyTorch's built-in implementation.         | You deploy on Blackwell GPUs.                                                                             | None on Blackwell. The platform does not know the GPU type, so it cannot set this for you.                                                                                 |

On Kubernetes, set `executor_config.disk_size` to fit the checkpoint, because the default `50Gi` [weights volume](#model-weights-volume) is too small: at least `300Gi` for BF16 and `100Gi` for NVFP4.

Send each image in the request in one of three forms:

* A `data:` URL that carries the image bytes inline: `data:image/png;base64,<base64-encoded bytes>` (or `image/jpeg`, `image/gif`, `image/webp`). The first example below builds one from a local file.
* A fileset reference, `fileset://<workspace>/<fileset>#<path>`, or `fileset://<fileset>#<path>` for a fileset in the request's workspace. The gateway downloads the file with your permissions and replaces the reference with a `data:` URL before the request reaches middleware or the model. The file must be PNG, JPEG, GIF, or WebP, as named by its extension, and at most 25 MiB. Fileset references work on every gateway route: by model entity, through the OpenAI-compatible endpoint, and by provider.
* An `http://` or `https://` URL. The model server downloads the image itself, so it needs network access to that host. To limit which hosts it downloads from, list them with `--allowed-media-domains`:

  ```json
  "additional_args": ["--allowed-media-domains", "upload.wikimedia.org", "images.example.com"]
  ```

  A URL on any other host is rejected. `data:` URLs and fileset references are not affected, because they reach the server as inline bytes.

Neither the gateway nor the model server reads local file paths, and the gateway rejects `file://` URLs.

#### CLI

```bash
nemo files filesets create "super-vl-bf16" \
  --storage '{
    "type": "huggingface",
    "repo_id": "nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16",
    "repo_type": "model"
  }'

# The checkpoint ships its own configuration code
nemo models create "super-vl-bf16" \
  --fileset "default/super-vl-bf16" \
  --trust-remote-code

nemo inference deployment-configs create "super-vl-config" \
  --engine "vllm" \
  --model-spec '{
    "model_namespace": "default",
    "model_name": "super-vl-bf16"
  }' \
  --executor-config '{
    "gpu": 4,
    "disk_size": "300Gi",
    "additional_args": [
      "--max-model-len", "114688",
      "--reasoning-parser", "nemotron_v3",
      "--tool-call-parser", "qwen3_coder",
      "--enable-auto-tool-choice",
      "--mamba-ssm-cache-dtype", "float32",
      "--limit-mm-per-prompt", "{\"image\": 1, \"video\": 1}",
      "--speculative-config", "{\"method\": \"mtp\", \"num_speculative_tokens\": 3}"
    ]
  }'

nemo inference deployments create "super-vl-deployment" \
  --config "super-vl-config"

nemo wait inference deployment super-vl-deployment
```

#### Python SDK

```python
import base64

client.files.create_fileset(
    body=CreateFilesetRequest(
        name="super-vl-bf16",
        storage=HuggingfaceStorageConfig(
            repo_id="nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16", repo_type="model"
        ),
    ),
    exist_ok=True,
)

# The checkpoint ships its own configuration code
client.models.create_model(
    body=CreateModelEntityRequest(
        name="super-vl-bf16", fileset="default/super-vl-bf16", trust_remote_code=True
    ),
)

client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name="super-vl-config",
        engine=Engine.VLLM,
        model_spec=ModelDeploymentConfigModelSpec(
            model_namespace="default",
            model_name="super-vl-bf16",
        ),
        executor_config=ContainerExecutorConfig(
            gpu=4,
            disk_size="300Gi",
            additional_args=[
                "--max-model-len", "114688",
                "--reasoning-parser", "nemotron_v3",
                "--tool-call-parser", "qwen3_coder",
                "--enable-auto-tool-choice",
                "--mamba-ssm-cache-dtype", "float32",
                "--limit-mm-per-prompt", '{"image": 1, "video": 1}',
                "--speculative-config", '{"method": "mtp", "num_speculative_tokens": 3}',
            ],
        ),
    ),
)

client.models.create_deployment(
    body=CreateModelDeploymentRequest(
        name="super-vl-deployment",
        config="super-vl-config",
    ),
)

client.models.wait_for_deployment_status(
    deployment_name="super-vl-deployment",
    desired_status="READY",
)

# Send the image bytes inline as a data: URL
with open("receipt.png", "rb") as f:
    image_url = "data:image/png;base64," + base64.b64encode(f.read()).decode()

response = InferenceGatewayClient.from_client(client).provider_post(
    name="super-vl-deployment",
    trailing_uri="v1/chat/completions",
    body=JsonBody({
        "model": "default/super-vl-bf16",
        "messages": [{
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": image_url}},
                {"type": "text", "text": "What is the total on this receipt?"},
            ],
        }],
        # Thinking tokens count toward max_tokens
        "max_tokens": 2048,
    }),
)

# Or reference an image stored in a fileset; the gateway inlines it for you
client.files.create_fileset(body=CreateFilesetRequest(name="receipts"), exist_ok=True)
with open("receipt.png", "rb") as f:
    client.files.upload_file(name="receipts", path="receipt.png", content=f.read())

response = InferenceGatewayClient.from_client(client).model_post(
    name="super-vl-bf16",
    trailing_uri="v1/chat/completions",
    body=JsonBody({
        "model": "default/super-vl-bf16",
        "messages": [{
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": "fileset://default/receipts#receipt.png"}},
                {"type": "text", "text": "What is the total on this receipt?"},
            ],
        }],
        "max_tokens": 2048,
    }),
)
```

---

## Deployment Cleanup

#### CLI

```bash
# Note: Deleting the deployment will free up its GPU(s) when complete
nemo inference deployments delete <deployment-name>
nemo wait inference deployment <deployment-name> --status DELETED
nemo inference deployment-configs delete <config-name>

# For external providers
nemo inference providers delete <provider-name>
nemo secrets delete <secret-name>
```

#### Python SDK

```python
# Note: Deleting the deployment will free up its GPU(s) when complete
client.models.delete_deployment(name="<deployment-name>")
client.models.wait_for_deployment_status(
    deployment_name="<deployment-name>",
    desired_status="DELETED",
)
client.models.delete_deployment_config(name="<config-name>")

# For external providers
client.models.delete_provider(name="<provider-name>")
client.secrets.delete_secret(name="<secret-name>")
```

---

## Multi-GPU Deployments

For larger models requiring multiple GPUs, parallelism configuration depends on the NIM type.

### Parallelism Strategies

* **Tensor Parallel (TP)**: Splits model layers across GPUs → best for latency
* **Pipeline Parallel (PP)**: Splits model depth across GPUs → best for throughput
* **Formula**: `gpu` = `tp_size` × `pp_size`

### Model-Specific NIMs

Model-specific NIMs (for example, `nvcr.io/nim/meta/llama-3.1-70b-instruct`) have TP/PP settings derived from manifest profiles in the container. Configure enough GPUs and the NIM selects the appropriate profile automatically.

### Model-Free NIM

The model-free NIM (`nvcr.io/nim/nvidia/vllm-model-free-nim:latest`), the platform's default NIM image, generates profiles at startup for tensor parallelism 1, 2, 4, or 8 and pipeline parallelism 1, 2, or 4, and selects one that fits the GPUs and memory it sees. The selected split need not use all `gpu` GPUs; for example, with `gpu: 3` no profile uses all three. To control the split, set `NIM_TENSOR_PARALLEL_SIZE` and `NIM_PIPELINE_PARALLEL_SIZE` in `additional_envs` so that their product equals `gpu`; these override the selected profile. See [Model Profiles and Selection](https://docs.nvidia.com/nim/large-language-models/latest/deployment/model-profiles-and-selection.html).

### Multi-LLM NIM

The multi-LLM NIM (`nvcr.io/nim/nvidia/llm-nim`) uses all GPUs for tensor parallelism by default (TP = `gpu`, PP = 1). To split the GPUs differently, set `NIM_TENSOR_PARALLEL_SIZE` and `NIM_PIPELINE_PARALLEL_SIZE` in `additional_envs` so that their product equals `gpu`.

This example deploys [Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct) across 2 GPUs using tensor parallelism. Setting `NIM_TENSOR_PARALLEL_SIZE` works on both the model-free NIM and the multi-LLM NIM.

#### CLI

```bash
# Create a fileset pointing to the HuggingFace model
# Qwen models are public — token_secret is optional
nemo files filesets create "qwen-2-5-14b" \
  --storage '{
"type": "huggingface",
"repo_id": "Qwen/Qwen2.5-14B-Instruct",
"repo_type": "model"
}'

# Register a model entity referencing the fileset
nemo models create "qwen-2-5-14b" \
  --fileset "default/qwen-2-5-14b"

# Create deployment config with 2 GPUs and tensor parallelism 2
nemo inference deployment-configs create "qwen-14b-config" \
  --engine "nim" \
  --model-spec '{
    "model_name": "default/qwen-2-5-14b"
}' \
  --executor-config '{
    "gpu": 2,
    "additional_envs": {"NIM_TENSOR_PARALLEL_SIZE": "2"}
}'

# Deploy
nemo inference deployments create "qwen-14b-deployment" \
  --config "qwen-14b-config"

nemo wait inference deployment qwen-14b-deployment

nemo chat -m default/qwen-2-5-14b 'Hello!' \
  --max-tokens 100
```

#### Python SDK

```python
# Create a fileset pointing to the HuggingFace model
# Qwen models are public — token_secret is optional
client.files.create_fileset(
    body=CreateFilesetRequest(
        name="qwen-2-5-14b",
        storage=HuggingfaceStorageConfig(repo_id="Qwen/Qwen2.5-14B-Instruct", repo_type="model"),
    ),
    exist_ok=True,
)

# Register a model entity referencing the fileset
client.models.create_model(
    body=CreateModelEntityRequest(name="qwen-2-5-14b", fileset="default/qwen-2-5-14b"),
)

# Create deployment config with 2 GPUs and tensor parallelism 2
config = client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name="qwen-14b-config",
        engine=Engine.NIM,
        model_spec=ModelDeploymentConfigModelSpec(model_name="default/qwen-2-5-14b"),
        executor_config=ContainerExecutorConfig(gpu=2, additional_envs={"NIM_TENSOR_PARALLEL_SIZE": "2"}),
    ),
).data()

# Deploy
deployment = client.models.create_deployment(
    body=CreateModelDeploymentRequest(name="qwen-14b-deployment", config="qwen-14b-config"),
).data()

client.models.wait_for_deployment_status(
    deployment_name="qwen-14b-deployment",
    desired_status="READY",
)

response = InferenceGatewayClient.from_client(client).model_post(
    name="qwen-2-5-14b",
    trailing_uri="v1/chat/completions",
    body=JsonBody({
        "model": "default/qwen-2-5-14b",
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 100,
    }),
)
```

### Custom Parallelism Configuration

For larger models requiring more GPUs, you can configure specific TP/PP splits using `additional_envs`. The formula is: `gpu` = `NIM_TENSOR_PARALLEL_SIZE` × `NIM_PIPELINE_PARALLEL_SIZE`.

#### CLI

```bash
nemo inference deployment-configs create "multi-gpu-custom-config" \
  --engine "nim" \
  --model-spec '{
    "model_name": "default/qwen-2-5-14b"
}' \
  --executor-config '{
    "gpu": 4,
    "additional_envs": {
        "NIM_TENSOR_PARALLEL_SIZE": "2",
        "NIM_PIPELINE_PARALLEL_SIZE": "2"
    }
}'
```

#### Python SDK

```python
config = client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name="multi-gpu-custom-config",
        engine=Engine.NIM,
        model_spec=ModelDeploymentConfigModelSpec(model_name="default/qwen-2-5-14b"),
        executor_config=ContainerExecutorConfig(
            gpu=4,
            additional_envs={
                "NIM_TENSOR_PARALLEL_SIZE": "2",
                "NIM_PIPELINE_PARALLEL_SIZE": "2",
            },
        ),
    ),
).data()
```

> **Tip**
>
> **Choosing Parallelism Strategy**
>
> * **TP=8, PP=1** (default): Lowest latency, best for real-time applications
> * **TP=4, PP=2**: Balanced latency and throughput
> * **TP=2, PP=4**: Highest throughput, best for batch processing
>
> For custom models, match deployment parallelism to training parallelism for optimal performance.

---

## Chat Templates and Tool Calling

Configure custom chat templates and tool calling for NIM deployments. These settings control how the model formats chat messages and handles function/tool calling.

For more information on chat templates, see the [Hugging Face chat templating guide](https://huggingface.co/docs/transformers/en/chat_templating) and the [NeMo chat templates documentation](https://docs.nvidia.com/nemo/rl/latest/design-docs/chat-datasets.html#chat-templates).

> **Warning**
>
> **Security consideration:** Chat templates are Jinja2 programs that execute on every inference call. While NIM uses a sandboxed Jinja2 environment (mitigating arbitrary code execution), a malicious or misconfigured template can still alter model behavior — for example, by injecting hidden instructions, rewriting messages, or degrading output quality. Grant chat template permissions only to trusted users, and review templates before deploying to production. See [Inference-Time Backdoors via Chat Templates (IEEE S\&P 2026)](https://arxiv.org/abs/2602.04653) for further background.

### Configuration Sources

There are two ways to configure chat templates and tool calling, with deployment-level settings taking highest priority:

| Source                                    | Priority | When to Use                                                                       |
| ----------------------------------------- | -------- | --------------------------------------------------------------------------------- |
| **Fileset `metadata.model.tool_calling`** | Base     | Set once per model — applies to all deployments using that model                  |
| **Deployment config (`model_spec`)**      | Override | Per-deployment overrides — useful for A/B testing or deployment-specific behavior |

When both are set, deployment config values override fileset values.

### How It Works

1. User sets `chat_template`, `tool_call_parser`, `tool_call_plugin`, and/or `auto_tool_choice` on the fileset via `metadata.model.tool_calling`
2. The model-spec background task reads the fileset's `metadata.model.tool_calling` and writes them into the model entity's `spec`
3. At deployment time, the platform reads `model_entity.spec` and any deployment-level overrides and passes them to NIM (see [NIM Configuration Mapping](#nim-configuration-mapping))

```mermaid
flowchart LR
 FS["Fileset<br />metadata.model.tool_calling"] -->|model-spec task| ME["Model Entity<br />spec"]
 ME --> ENV["NIM Container<br />env vars"]
 DC["Deployment Config<br />model_spec"] -->|override| ENV
```

### Option 1: Set via Fileset (Recommended)

Set `chat_template` and tool calling configuration with fileset `metadata.model.tool_calling`. The platform automatically propagates these into the model entity spec when the model-spec background task runs.

#### CLI

```bash

 # Create fileset with chat template and tool calling config via metadata
 nemo files filesets create "llama-3-2-1b-tool" \
 --storage '{
 "type": "huggingface",
 "repo_id": "meta-llama/Llama-3.2-1B-Instruct",
 "repo_type": "model"
 }' \
 --metadata '{
 "model": {
 "tool_calling": {
 "chat_template": "{%- for message in messages %}{%- set content = '\''<{{ start_header_id }}>'\'' + message['\''role'\''] + '\''<{{ end_header_id }}>\n\n'\'' + message['\''content'\''] | trim + '\''<{{ eot_id }}>'\'' %}{%- if loop.index0 == 0 %}{%- set content = '\''<{{ begin_of_text }}>'\'' + content %}{%- endif %}{{ content }}{%- endfor %}{%- if add_generation_prompt %}{{ '\''<{{ start_header_id }}>assistant<{{ end_header_id }}>\n\n'\'' }}{%- endif %}",
 "tool_call_parser": "llama3_json",
 "auto_tool_choice": true
 }
 }
 }'

 # Register model entity referencing the fileset
 nemo models create "llama-3-2-1b-tool" \
 --fileset "default/llama-3-2-1b-tool"

 # Deploy
 nemo inference deployment-configs create "llama-tool-config" \
 --engine "nim" \
 --model-spec '{
    "model_name": "default/llama-3-2-1b-tool"
 }' \
 --executor-config '{
    "gpu": 1
 }'

 nemo inference deployments create "llama-tool-deployment" \
 --config "llama-tool-config"

 nemo wait inference deployment llama-tool-deployment

```

#### Python SDK

```python

# Create fileset with chat template and tool calling config via metadata
tool_metadata = FilesetMetadata(
    model=ModelMetadataContent(
        tool_calling=ToolCallingMetadataContent(
            chat_template=(
                "{%- for message in messages %}"
                "{%- set content = '<{{ start_header_id }}>' + message['role'] + '<{{ end_header_id }}>\n\n'"
                " + message['content'] | trim + '<{{ eot_id }}>' %}"
                "{%- if loop.index0 == 0 %}{%- set content = '<{{ begin_of_text }}>' + content %}{%- endif %}"
                "{{ content }}{%- endfor %}"
                "{%- if add_generation_prompt %}{{ '<{{ start_header_id }}>assistant<{{ end_header_id }}>\n\n' }}{%- endif %}"
            ),
            tool_call_parser="llama3_json",
            auto_tool_choice=True,
        )
    )
)

client.files.create_fileset(
    body=CreateFilesetRequest(
        name="llama-3-2-1b-tool",
        storage=HuggingfaceStorageConfig(
            repo_id="meta-llama/Llama-3.2-1B-Instruct",
            repo_type="model",
        ),
        metadata=tool_metadata,
    ),
    exist_ok=True,
)

# Register model entity referencing the fileset
client.models.create_model(
    body=CreateModelEntityRequest(
        name="llama-3-2-1b-tool",
        fileset="default/llama-3-2-1b-tool",
    ),
)

# Deploy — chat_template and tool_call_config are inherited from the fileset
config = client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name="llama-tool-config",
        engine=Engine.NIM,
        model_spec=ModelDeploymentConfigModelSpec(model_name="default/llama-3-2-1b-tool"),
        executor_config=ContainerExecutorConfig(gpu=1),
    ),
).data()

deployment = client.models.create_deployment(
    body=CreateModelDeploymentRequest(
        name="llama-tool-deployment",
        config="llama-tool-config",
    ),
).data()

client.models.wait_for_deployment_status(
    deployment_name="llama-tool-deployment",
    desired_status="READY",
)

```

### Option 2: Set via Deployment Config (Override)

Set `chat_template` and `tool_call_config` directly on the deployment config. These override any values from the fileset.

#### CLI

```bash

 nemo inference deployment-configs create "llama-tool-override-config" \
 --engine "nim" \
 --model-spec '{
    "model_name": "default/llama-3-2-1b-tool",
    "chat_template": "{%- for message in messages %}{{ message[\"content\"] }}{%- endfor %}",
    "tool_call_config": {
        "tool_call_parser": "hermes",
        "auto_tool_choice": false
    }
 }' \
 --executor-config '{
    "gpu": 1
 }'

```

#### Python SDK

```python

config = client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name="llama-tool-override-config",
        engine=Engine.NIM,
        model_spec=ModelDeploymentConfigModelSpec(
            model_name="default/llama-3-2-1b-tool",
            chat_template="{%- for message in messages %}{{ message['content'] }}{%- endfor %}",
            tool_call_config=ToolCallConfig(tool_call_parser="hermes", auto_tool_choice=False),
        ),
        executor_config=ContainerExecutorConfig(gpu=1),
    ),
).data()

```

### Change Tool Calling Config for an Existing Model

Updating a fileset's `metadata.model.tool_calling` does **not** propagate changes to an existing model entity. The model entity's `spec` is populated from the fileset only at creation time. To change the tool calling configuration, create a new fileset with the updated config and a new model entity that references it.

#### CLI

```bash
# Create a new fileset with the updated tool calling config
nemo files filesets create "llama-3-2-1b-tool-v2" \
  --storage '{
"type": "huggingface",
"repo_id": "meta-llama/Llama-3.2-1B-Instruct",
"repo_type": "model"
}' \
  --metadata '{
"model": {
"tool_calling": {
"tool_call_parser": "mistral",
"auto_tool_choice": true
}
}
}'

# Create a new model entity referencing the new fileset
nemo models create "llama-3-2-1b-tool-v2" \
  --fileset "default/llama-3-2-1b-tool-v2"

# Update deployment config to use the new model, or create a new one
nemo inference deployment-configs create "llama-tool-config-v2" \
  --engine "nim" \
  --model-spec '{
    "model_name": "default/llama-3-2-1b-tool-v2"
}' \
  --executor-config '{
    "gpu": 1
}'

nemo inference deployments create "llama-tool-deployment-v2" \
  --config "llama-tool-config-v2"

nemo wait inference deployment llama-tool-deployment-v2
```

#### Python SDK

```python
# Create a new fileset with the updated tool calling config via metadata
tool_metadata_v2 = FilesetMetadata(
    model=ModelMetadataContent(
        tool_calling=ToolCallingMetadataContent(
            tool_call_parser="mistral",
            auto_tool_choice=True,
        )
    )
)

client.files.create_fileset(
    body=CreateFilesetRequest(
        name="llama-3-2-1b-tool-v2",
        storage=HuggingfaceStorageConfig(
            repo_id="meta-llama/Llama-3.2-1B-Instruct",
            repo_type="model",
        ),
        metadata=tool_metadata_v2,
    ),
    exist_ok=True,
)

# Create a new model entity referencing the new fileset
client.models.create_model(
    body=CreateModelEntityRequest(
        name="llama-3-2-1b-tool-v2",
        fileset="default/llama-3-2-1b-tool-v2",
    ),
)

# Create a new deployment config and deployment
config = client.models.create_deployment_config(
    body=CreateModelDeploymentConfigRequest(
        name="llama-tool-config-v2",
        engine=Engine.NIM,
        model_spec=ModelDeploymentConfigModelSpec(model_name="default/llama-3-2-1b-tool-v2"),
        executor_config=ContainerExecutorConfig(gpu=1),
    ),
).data()

deployment = client.models.create_deployment(
    body=CreateModelDeploymentRequest(
        name="llama-tool-deployment-v2",
        config="llama-tool-config-v2",
    ),
).data()

client.models.wait_for_deployment_status(
    deployment_name="llama-tool-deployment-v2",
    desired_status="READY",
)
```

### Custom Tool Call Plugin

For custom tool calling parsers, store the plugin Python file in a separate fileset and reference it via `tool_call_plugin`.

> **Note**
>
> Because plugins execute arbitrary Python code inside the NIM container, `tool_call_plugin` is **disabled by default** at the platform level.
> To enable it, set `models.tool_call_plugin.enabled: true` in the platform configuration and ensure the user has the `models.tool-call-plugin.set` permission (granted to Admin and HelixAdmin roles by default).

#### Python SDK

```python
# 1. Create a fileset for the plugin file
client.files.create_fileset(body=CreateFilesetRequest(name="my-tool-plugin"), exist_ok=True)
client.files.upload_file(
    name="my-tool-plugin",
    path="my_parser.py",
    content=Path("my_parser.py").read_bytes(),
)

# 2. Reference the plugin fileset when creating a new model fileset.
# Updating metadata on a fileset that already backs an existing model entity
# does not update that model entity's spec.
plugin_tool_metadata = FilesetMetadata(
    model=ModelMetadataContent(
        tool_calling=ToolCallingMetadataContent(
            tool_call_parser="custom_parser",
            tool_call_plugin="default/my-tool-plugin",
            auto_tool_choice=True,
        )
    )
)

client.files.create_fileset(
    body=CreateFilesetRequest(
        name="llama-3-2-1b-tool-plugin",
        storage=HuggingfaceStorageConfig(
            repo_id="meta-llama/Llama-3.2-1B-Instruct",
            repo_type="model",
        ),
        metadata=plugin_tool_metadata,
    ),
    exist_ok=True,
)
client.models.create_model(
    body=CreateModelEntityRequest(
        name="llama-3-2-1b-tool-plugin",
        fileset="default/llama-3-2-1b-tool-plugin",
    ),
)
```

The platform downloads the plugin fileset at deployment time and passes the `.py` file path to NIM: through the `NIM_TOOL_PARSER_PLUGIN` environment variable on NIM 1.x images, and through the `--tool-parser-plugin` flag on the vLLM model-free NIM. The SGLang model-free NIM does not support parser plugins.

### Tool Call Config Reference

| Field              | Type      | Description                                                                                    |
| ------------------ | --------- | ---------------------------------------------------------------------------------------------- |
| `tool_call_parser` | `string`  | Parser name: `llama3_json`, `hermes`, `mistral`, `pythonic`, `openai`, or a custom parser name |
| `tool_call_plugin` | `string`  | Fileset reference (`workspace/fileset-name`) containing a custom plugin `.py` file             |
| `auto_tool_choice` | `boolean` | When `true`, the model can decide to call tools without explicit user instruction              |

### NIM Configuration Mapping

NIM 1.x images read tool calling configuration from environment variables. The model-free NIM (NIM 2.x) does not read those variables; it takes backend flags in `NIM_PASSTHROUGH_ARGS` instead. The platform sets whichever the deployment's image reads:

| Metadata Field                  | NIM 1.x Environment Variable                        | Model-Free NIM Flag (in `NIM_PASSTHROUGH_ARGS`)                                                         |
| ------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `tool_calling.chat_template`    | `NIM_CHAT_TEMPLATE` (Jinja2 template string)        | `--chat-template` (the platform writes the template to a file in the container)                         |
| `tool_calling.tool_call_parser` | `NIM_TOOL_CALL_PARSER` (for example, `llama3_json`) | `--tool-call-parser`                                                                                    |
| `tool_calling.tool_call_plugin` | `NIM_TOOL_PARSER_PLUGIN` (plugin file path)         | `--tool-parser-plugin` (vLLM only; the deployment fails on the SGLang model-free NIM)                   |
| `tool_calling.auto_tool_choice` | `NIM_ENABLE_AUTO_TOOL_CHOICE` (`1` or `0`)          | `--enable-auto-tool-choice` (when `true`; not set on the SGLang model-free NIM, which rejects the flag) |

On the model-free NIM, the platform places these flags before any `NIM_PASSTHROUGH_ARGS` you set in `executor_config.additional_envs`. NIM keeps the last value of a repeated flag, so your own flags take precedence. Parser names differ between vLLM and SGLang, so set a `tool_call_parser` that the image's engine supports.

---

## Controlling Reasoning at Request Level

Some models (for example, `nvidia/nemotron-3.5-lightning-30b-a3b`) enable reasoning/thinking by default. (From the model card: "\[nemotron-3.5] responds to user queries and tasks by first generating a reasoning trace and then concluding with a final response") You can disable it on a per-request basis by passing `chat_template_kwargs` directly in the request body:

#### CLI

```bash
nemo inference gateway provider post v1/chat/completions "my-deployment" \
  --body '{
"model": "default/my-model",
"messages": [{"role": "user", "content": "Hello!"}],
"max_tokens": 100,
"chat_template_kwargs": {"enable_thinking": false}
}'
```

#### Python SDK

```python
response = InferenceGatewayClient.from_client(client).provider_post(
    name="my-deployment",
    trailing_uri="v1/chat/completions",
    body=JsonBody({
        "model": "default/my-model",
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 100,
        "chat_template_kwargs": {"enable_thinking": False},
    }),
)
```

> **Note**
>
> `chat_template_kwargs` is passed directly in the request body, not nested under `extra_body`. For more details on request-level reasoning overrides, see the [vLLM documentation](https://docs.vllm.ai/en/latest/features/reasoning_outputs/#request-level-override).

This parameter only applies to models which use vLLM under the hood. For non-vLLM providers such as OpenAI or NVIDIA Build, the parameter that controls reasoning differs. Consult the provider's documentation for the specific parameter.