Deploy Models | NVIDIA NeMo Platform

Deploy models from NGC or HuggingFace. Register external providers like OpenAI or NVIDIA Build.

Resource names for deployments, deployment configs, and providers must contain only letters (a-z, A-Z), digits (0-9), underscores, hyphens, and dots. For example: llama-3.1-8b, my-custom-model, qwen-fs-config.

CLI

Python SDK

$ # Configure CLI (if not already done)
$ nemo config set --base-url "$NMP_BASE_URL" --workspace default

Add External Providers

NVIDIA Build

By default, the platform pre-configures an external provider for NVIDIA Build named nvidia-build in the system workspace. The example below demonstrates how to recreate it in your own workspace. For disambiguation purposes, this example names the manually-created version my-nvidia-build.

CLI

Python SDK

$ # Store API key
$ echo "$NVIDIA_API_KEY" | nemo secrets create "nvidia-api-key" --from-file -
$ 
$ # Create provider
$ nemo inference providers create "my-nvidia-build" \
> --host-url "https://integrate.api.nvidia.com" \
> --api-key-secret-name "nvidia-api-key"
$ 
$ nemo wait inference provider my-nvidia-build
$ 
$ # Test using interactive chat
$ nemo chat nvidia/llama-3.3-nemotron-super-49b-v1 'Hello!' \
> --provider my-nvidia-build

OpenAI

CLI

Python SDK

$ # Store API key
$ echo "$OPENAI_API_KEY" | nemo secrets create "openai-api-key" --from-file -
$ 
$ # Create provider with enabled models
$ nemo inference providers create "openai" \
> --host-url "https://api.openai.com/v1" \
> --api-key-secret-name "openai-api-key" \
> --enabled-models "gpt-4" \
> --enabled-models "gpt-3.5-turbo"
$ 
$ nemo wait inference provider openai
$ 
$ # Test using interactive chat
$ nemo chat gpt-4 'Hello!' \
> --provider openai

Anthropic

Anthropic’s /v1/messages API expects the API key in an X-Api-Key: header (not Authorization: Bearer) and requires an anthropic-version header on every request. Use --auth-header-format (Jinja2 template, must contain exactly one {{ auth_secret }} variable) to override the default Authorization: Bearer {{ auth_secret }} and pass the API-version pin via --default-extra-headers. Without these, Anthropic rejects every request with 401.

CLI

Python SDK

$ # Store API key
$ echo "$ANTHROPIC_API_KEY" | nemo secrets create "anthropic-api-key" --from-file -
$ 
$ # Create provider — override the default Bearer header and pin the API version
$ nemo inference providers create "anthropic" \
> --host-url "https://api.anthropic.com" \
> --api-key-secret-name "anthropic-api-key" \
> --auth-header-format "X-Api-Key: {{ auth_secret }}" \
> --default-extra-headers '{"anthropic-version": "2023-06-01"}'
$ 
$ nemo wait inference provider anthropic

{{ auth_secret }} is substituted with the resolved secret value at request time.

Deploy from NGC

Deploy pre-built NIM containers from NGC.

Deploy Llama 3.2 1B

CLI

Python SDK

$ nemo inference deployment-configs create \
> --name "llama-3-2-1b-config" \
> --engine "nim" \
> --model-spec '{
>     "model_name": "meta/llama-3.2-1b-instruct"
> }' \
> --executor-config '{
>     "gpu": 1,
>     "image_name": "nvcr.io/nim/meta/llama-3.2-1b-instruct",
>     "image_tag": "1.8.6"
> }'
$ 
$ nemo inference deployments create \
> --name "llama-3-2-1b-deployment" \
> --config "llama-3-2-1b-config"
$ 
$ nemo wait inference deployment llama-3-2-1b-deployment
$ 
$ nemo chat meta/llama-3.2-1b-instruct 'Hello!' \
> --provider llama-3-2-1b-deployment \
> --max-tokens 100

Deploy NemoGuard JailbreakDetect

Deploy classification NIMs like NemoGuard for content safety. Uses the /v1/classify endpoint instead of chat completions.

CLI

Python SDK

$ nemo inference deployment-configs create \
> --name "nemoguard-jailbreak-config" \
> --engine "nim" \
> --model-spec '{}' \
> --executor-config '{
>     "gpu": 1,
>     "image_name": "nvcr.io/nim/nvidia/nemoguard-jailbreak-detect",
>     "image_tag": "1.10.1"
> }'
$ 
$ nemo inference deployments create \
> --name "nemoguard-jailbreak-deployment" \
> --config "nemoguard-jailbreak-config"
$ 
$ nemo wait inference deployment nemoguard-jailbreak-deployment
$ 
$ nemo inference gateway provider post v1/classify \
> --name "nemoguard-jailbreak-deployment" \
> --body '{"input": "Tell me about vacation spots in Hawaii."}'

Deploy from HuggingFace

HuggingFace deployments use the Multi-LLM NIM (nvcr.io/nim/nvidia/llm-nim:1.13.1) by default, which only supports specific model architectures. Check the supported architectures list before deploying. If your model architecture is not listed, you will need a model-specific NIM image — see Deploy from NGC for that approach.

You can register a HuggingFace model through the Files service. This creates a fileset that acts as a proxy. The Files service handles authentication and caches the weights on first download, so subsequent deployments start faster.

CLI

Python SDK

$ # (Optional) Create a HuggingFace token secret for private models.
$ # Public models like Qwen do not require a token.
$ echo "$HF_TOKEN" | nemo secrets create "hf-token-secret" --from-file -
$ 
$ # Create a fileset pointing to the HuggingFace model.
$ # "token_secret" is optional — only needed for private/gated models.
$ nemo files filesets create "qwen-2-5-1-5b" \
> --storage '{
> "type": "huggingface",
> "repo_id": "Qwen/Qwen2.5-1.5B-Instruct",
> "repo_type": "model",
> "token_secret": "hf-token-secret"
> }'
$ 
$ # Register a model entity referencing the fileset
$ nemo models create "qwen-2-5-1-5b" \
> --fileset "default/qwen-2-5-1-5b"
$ 
$ # Create deployment config pointing to the model entity
$ nemo inference deployment-configs create "qwen-fs-config" \
> --engine "nim" \
> --model-spec '{
>     "model_namespace": "default",
>     "model_name": "qwen-2-5-1-5b"
> }' \
> --executor-config '{
>     "gpu": 1
> }'
$ 
$ nemo inference deployments create "qwen-fs-deployment" \
> --config "qwen-fs-config"
$ 
$ nemo wait inference deployment qwen-fs-deployment
$ 
$ nemo chat default/qwen-2-5-1-5b 'Hello!' \
> --provider qwen-fs-deployment \
> --max-tokens 100

The fileset format is <workspace>/<fileset-name>. This tells the deployment system to pull weights from the Files service, which proxies the download from HuggingFace using the fileset token_secret. For public models like Qwen, the token_secret on the fileset is optional.

Deploy with vLLM

Serve any HuggingFace-format model with vLLM by setting engine: "vllm" on the deployment config. The model is registered through the Files service exactly as in Deploy from HuggingFace — only the engine field on the deployment config changes. Tensor parallelism is computed automatically from the GPU count and the model’s architecture.

This example deploys Qwen3-1.7B with vLLM.

CLI

Python SDK

$ # Create a fileset pointing to the HuggingFace model.
$ # Qwen models are public — token_secret is optional.
$ nemo files filesets create "qwen3-1-7b" \
>   --storage '{
>     "type": "huggingface",
>     "repo_id": "Qwen/Qwen3-1.7B",
>     "repo_type": "model"
>   }'
$ 
$ # Register a model entity referencing the fileset
$ nemo models create "qwen3-1-7b" \
>   --fileset "default/qwen3-1-7b"
$ 
$ # Create deployment config with engine "vllm"
$ nemo inference deployment-configs create "qwen3-vllm-config" \
>   --engine "vllm" \
>   --model-spec '{
>     "model_namespace": "default",
>     "model_name": "qwen3-1-7b"
>   }' \
>   --executor-config '{
>     "gpu": 1
>   }'
$ 
$ nemo inference deployments create "qwen3-vllm-deployment" \
>   --config "qwen3-vllm-config"
$ 
$ nemo wait inference deployment qwen3-vllm-deployment
$ 
$ nemo chat default/qwen3-1-7b 'Hello!' \
>   --provider qwen3-vllm-deployment \
>   --max-tokens 100

Deploy with LoRA Adapters

vLLM deployments support LoRA adapters with hot-reload: enable LoRA on the deployment config, then register one or more adapters against the base model entity. An adapter sidecar delivers each enabled adapter to the running server, and vLLM serves it on a per-request basis. Adapters are registered on the base model entity, not as separate models.

This example serves the linear-algebra LoRA on top of Qwen3-1.7B.

CLI

Python SDK

$ # Base model fileset + entity (skip if already created above)
$ nemo files filesets create "qwen3-1-7b" \
>   --storage '{
>     "type": "huggingface",
>     "repo_id": "Qwen/Qwen3-1.7B",
>     "repo_type": "model"
>   }'
$ 
$ nemo models create "qwen3-1-7b" \
>   --fileset "default/qwen3-1-7b"
$ 
$ # Fileset for the LoRA adapter weights
$ nemo files filesets create "linear-algebra-lora" \
>   --storage '{
>     "type": "huggingface",
>     "repo_id": "premjatin/qwen-linear-algebra-coder",
>     "repo_type": "model"
>   }'
$ 
$ # Deployment config with LoRA enabled.
$ # Set --max-lora-rank to the highest adapter rank you intend to serve.
$ nemo inference deployment-configs create "qwen3-lora-config" \
>   --engine "vllm" \
>   --model-spec '{
>     "model_namespace": "default",
>     "model_name": "qwen3-1-7b",
>     "lora_enabled": true
>   }' \
>   --executor-config '{
>     "gpu": 1,
>     "additional_args": ["--max-lora-rank", "16"]
>   }'
$ 
$ nemo inference deployments create "qwen3-lora-deployment" \
>   --config "qwen3-lora-config"
$ 
$ nemo wait inference deployment qwen3-lora-deployment
$ 
$ # Register the adapter against the base model entity.
$ # The adapter is served under the name "<workspace>--<adapter-name>".
$ nemo models adapters create "qwen3-1-7b" "linear-algebra" \
>   --fileset "default/linear-algebra-lora" \
>   --finetuning-type lora \
>   --enabled
$ 
$ # Target the adapter by its served name in the request body
$ nemo inference gateway provider post v1/chat/completions qwen3-lora-deployment \
>   --body '{
>     "model": "default--linear-algebra",
>     "messages": [{"role": "user", "content": "Find the determinant of [[2, 1], [1, 3]]."}],
>     "max_tokens": 256
>   }'

The default vLLM image is configurable by the platform operator. To override the image per deployment, set image_name / image_tag on executor_config. To pass raw vllm serve flags (for example --max-model-len), use additional_args on executor_config.

Deployment Cleanup

CLI

Python SDK

$ # Note: Deleting the deployment will free up its GPU(s) when complete
$ nemo inference deployments delete <deployment-name>
$ nemo wait inference deployment <deployment-name> --status DELETED
$ nemo inference deployment-configs delete <config-name>
$ 
$ # For external providers
$ nemo inference providers delete <provider-name>
$ nemo secrets delete <secret-name>

Multi-GPU Deployments

For larger models requiring multiple GPUs, parallelism configuration depends on the NIM type.

Parallelism Strategies

Tensor Parallel (TP): Splits model layers across GPUs → best for latency
Pipeline Parallel (PP): Splits model depth across GPUs → best for throughput
Formula: gpu = tp_size × pp_size

Model-Specific NIMs

Model-specific NIMs (for example, nvcr.io/nim/meta/llama-3.1-70b-instruct) have TP/PP settings derived from manifest profiles in the container. Configure enough GPUs and the NIM selects the appropriate profile automatically.

Multi-LLM NIM

The multi-LLM NIM (nvcr.io/nim/nvidia/llm-nim:1.13.1) requires explicit parallelism configuration via environment variables (NIM_TENSOR_PARALLEL_SIZE, NIM_PIPELINE_PARALLEL_SIZE). By default, it uses all GPUs for tensor parallelism (TP=gpu, PP=1).

This example deploys Qwen2.5-14B-Instruct across 2 GPUs using tensor parallelism.

CLI

Python SDK

$ # Create a fileset pointing to the HuggingFace model
$ # Qwen models are public — token_secret is optional
$ nemo files filesets create \
> --name "qwen-2-5-14b" \
> --storage '{
> "type": "huggingface",
> "repo_id": "Qwen/Qwen2.5-14B-Instruct",
> "repo_type": "model"
> }'
$ 
$ # Register a model entity referencing the fileset
$ nemo models create \
> --name "qwen-2-5-14b" \
> --fileset "default/qwen-2-5-14b"
$ 
$ # Create deployment config with 2 GPUs (TP=2 by default)
$ nemo inference deployment-configs create \
> --name "qwen-14b-config" \
> --engine "nim" \
> --model-spec '{
>     "model_name": "default/qwen-2-5-14b"
> }' \
> --executor-config '{
>     "gpu": 2
> }'
$ 
$ # Deploy
$ nemo inference deployments create \
> --name "qwen-14b-deployment" \
> --config "qwen-14b-config"
$ 
$ nemo wait inference deployment qwen-14b-deployment
$ 
$ nemo chat default/qwen-2-5-14b 'Hello!' \
> --max-tokens 100

Custom Parallelism Configuration

For larger models requiring more GPUs, you can configure specific TP/PP splits using additional_envs. The formula is: gpu = NIM_TENSOR_PARALLEL_SIZE × NIM_PIPELINE_PARALLEL_SIZE.

CLI

Python SDK

$ nemo inference deployment-configs create \
> --name "multi-gpu-custom-config" \
> --engine "nim" \
> --model-spec '{
>     "model_name": "default/qwen-2-5-14b"
> }' \
> --executor-config '{
>     "gpu": 4,
>     "additional_envs": {
>         "NIM_TENSOR_PARALLEL_SIZE": "2",
>         "NIM_PIPELINE_PARALLEL_SIZE": "2"
>     }
> }'

Choosing Parallelism Strategy

TP=8, PP=1 (default): Lowest latency, best for real-time applications
TP=4, PP=2: Balanced latency and throughput
TP=2, PP=4: Highest throughput, best for batch processing

For custom models, match deployment parallelism to training parallelism for optimal performance.

Chat Templates and Tool Calling

Configure custom chat templates and tool calling for NIM deployments. These settings control how the model formats chat messages and handles function/tool calling.

For more information on chat templates, see the Hugging Face chat templating guide and the NeMo chat templates documentation.

Security consideration: Chat templates are Jinja2 programs that execute on every inference call. While NIM uses a sandboxed Jinja2 environment (mitigating arbitrary code execution), a malicious or misconfigured template can still alter model behavior — for example, by injecting hidden instructions, rewriting messages, or degrading output quality. Grant chat template permissions only to trusted users, and review templates before deploying to production. See Inference-Time Backdoors via Chat Templates (IEEE S&P 2026) for further background.

Configuration Sources

There are two ways to configure chat templates and tool calling, with deployment-level settings taking highest priority:

Source	Priority	When to Use
Fileset `metadata.model.tool_calling`	Base	Set once per model — applies to all deployments using that model
Deployment config (`model_spec`)	Override	Per-deployment overrides — useful for A/B testing or deployment-specific behavior

When both are set, deployment config values override fileset values.

How It Works

User sets chat_template, tool_call_parser, tool_call_plugin, and/or auto_tool_choice on the fileset via metadata.model.tool_calling
The model-spec background task reads the fileset’s metadata.model.tool_calling and writes them into the model entity’s spec
At deployment time, the platform reads model_entity.spec and any deployment-level overrides to set NIM environment variables

Option 1: Set via Fileset (Recommended)

Set chat_template and tool calling configuration with fileset metadata.model.tool_calling. The platform automatically propagates these into the model entity spec when the model-spec background task runs.

CLI

Python SDK

$  # Create fileset with chat template and tool calling config via metadata
$  nemo files filesets create \
>  --name "llama-3-2-1b-tool" \
>  --storage '{
>  "type": "huggingface",
>  "repo_id": "meta-llama/Llama-3.2-1B-Instruct",
>  "repo_type": "model"
>  }' \
>  --metadata '{
>  "model": {
>  "tool_calling": {
>  "chat_template": "{%- for message in messages %}{%- set content = '\''<{{ start_header_id }}>'\'' + message['\''role'\''] + '\''<{{ end_header_id }}>\n\n'\'' + message['\''content'\''] | trim + '\''<{{ eot_id }}>'\'' %}{%- if loop.index0 == 0 %}{%- set content = '\''<{{ begin_of_text }}>'\'' + content %}{%- endif %}{{ content }}{%- endfor %}{%- if add_generation_prompt %}{{ '\''<{{ start_header_id }}>assistant<{{ end_header_id }}>\n\n'\'' }}{%- endif %}",
>  "tool_call_parser": "llama3_json",
>  "auto_tool_choice": true
>  }
>  }
>  }'
$ 
$  # Register model entity referencing the fileset
$  nemo models create \
>  --name "llama-3-2-1b-tool" \
>  --fileset "default/llama-3-2-1b-tool"
$ 
$  # Deploy
$  nemo inference deployment-configs create \
>  --name "llama-tool-config" \
>  --engine "nim" \
>  --model-spec '{
>     "model_name": "default/llama-3-2-1b-tool"
>  }' \
>  --executor-config '{
>     "gpu": 1
>  }'
$ 
$  nemo inference deployments create \
>  --name "llama-tool-deployment" \
>  --config "llama-tool-config"
$ 
$  nemo wait inference deployment llama-tool-deployment

Option 2: Set via Deployment Config (Override)

Set chat_template and tool_call_config directly on the deployment config. These override any values from the fileset.

CLI

Python SDK

$  nemo inference deployment-configs create \
>  --name "llama-tool-override-config" \
>  --engine "nim" \
>  --model-spec '{
>     "model_name": "default/llama-3-2-1b-tool",
>     "chat_template": "{%- for message in messages %}{{ message[\"content\"] }}{%- endfor %}",
>     "tool_call_config": {
>         "tool_call_parser": "hermes",
>         "auto_tool_choice": false
>     }
>  }' \
>  --executor-config '{
>     "gpu": 1
>  }'

Change Tool Calling Config for an Existing Model

Updating a fileset’s metadata.model.tool_calling does not propagate changes to an existing model entity. The model entity’s spec is populated from the fileset only at creation time. To change the tool calling configuration, create a new fileset with the updated config and a new model entity that references it.

CLI

Python SDK

$ # Create a new fileset with the updated tool calling config
$ nemo files filesets create \
> --name "llama-3-2-1b-tool-v2" \
> --storage '{
> "type": "huggingface",
> "repo_id": "meta-llama/Llama-3.2-1B-Instruct",
> "repo_type": "model"
> }' \
> --metadata '{
> "model": {
> "tool_calling": {
> "tool_call_parser": "mistral",
> "auto_tool_choice": true
> }
> }
> }'
$ 
$ # Create a new model entity referencing the new fileset
$ nemo models create \
> --name "llama-3-2-1b-tool-v2" \
> --fileset "default/llama-3-2-1b-tool-v2"
$ 
$ # Update deployment config to use the new model, or create a new one
$ nemo inference deployment-configs create \
> --name "llama-tool-config-v2" \
> --engine "nim" \
> --model-spec '{
>     "model_name": "default/llama-3-2-1b-tool-v2"
> }' \
> --executor-config '{
>     "gpu": 1
> }'
$ 
$ nemo inference deployments create \
> --name "llama-tool-deployment-v2" \
> --config "llama-tool-config-v2"
$ 
$ nemo wait inference deployment llama-tool-deployment-v2

Custom Tool Call Plugin

For custom tool calling parsers, store the plugin Python file in a separate fileset and reference it via tool_call_plugin.

Because plugins execute arbitrary Python code inside the NIM container, tool_call_plugin is disabled by default at the platform level. To enable it, set models.tool_call_plugin.enabled: true in the platform configuration and ensure the user has the models.tool-call-plugin.set permission (granted to Admin and PlatformAdmin roles by default).

Python SDK

1 # 1. Create a fileset for the plugin file
2 client.files.filesets.create(name="my-tool-plugin")
3 client.files.upload(
4     fileset="my-tool-plugin",
5     local_path="my_parser.py",
6     remote_path="my_parser.py",
7 )
8 
9 # 2. Reference the plugin fileset in the model's fileset metadata
10 plugin_tool_metadata: dict[str, object] = {
11     "model": {
12         "tool_calling": {
13             "tool_call_parser": "custom_parser",
14             "tool_call_plugin": "default/my-tool-plugin",
15             "auto_tool_choice": True,
16         }
17     }
18 }
19 
20 client.files.filesets.update(
21     "llama-3-2-1b-tool",
22     metadata=plugin_tool_metadata,
23 )

The platform downloads the plugin fileset at deployment time and passes the .py file path to NIM via the NIM_TOOL_PARSER_PLUGIN environment variable.

Tool Call Config Reference

Field	Type	Description
`tool_call_parser`	`string`	Parser name: `llama3_json`, `hermes`, `mistral`, `pythonic`, `openai`, or a custom parser name
`tool_call_plugin`	`string`	Fileset reference (`workspace/fileset-name`) containing a custom plugin `.py` file
`auto_tool_choice`	`boolean`	When `true`, the model can decide to call tools without explicit user instruction

NIM Environment Variable Mapping

The platform translates tool calling configuration into NIM environment variables:

Metadata Field	NIM Environment Variable	Example Value
`tool_calling.chat_template`	`NIM_CHAT_TEMPLATE`	Jinja2 template string
`tool_calling.tool_call_parser`	`NIM_TOOL_CALL_PARSER`	`llama3_json`
`tool_calling.tool_call_plugin`	`NIM_TOOL_PARSER_PLUGIN`	`/path/to/my_parser.py`
`tool_calling.auto_tool_choice`	`NIM_ENABLE_AUTO_TOOL_CHOICE`	`1` or `0`

Controlling Reasoning at Request Level

Some models (for example, nvidia/nemotron-3-nano-30b-a3b) enable reasoning/thinking by default. (From the model card: “[nemotron-3] responds to user queries and tasks by first generating a reasoning trace and then concluding with a final response”) You can disable it on a per-request basis by passing chat_template_kwargs directly in the request body:

CLI

Python SDK

$ nemo inference gateway provider post v1/chat/completions \
> --name "my-deployment" \
> --body '{
> "model": "default/my-model",
> "messages": [{"role": "user", "content": "Hello!"}],
> "max_tokens": 100,
> "chat_template_kwargs": {"thinking": false}
> }'

chat_template_kwargs is passed directly in the request body, not nested under extra_body. For more details on request-level reasoning overrides, see the vLLM documentation.

This parameter only applies to models which use vLLM under the hood. For non-vLLM providers such as OpenAI or NVIDIA Build, the parameter that controls reasoning differs. Consult the provider’s documentation for the specific parameter.