Is this page helpful?

Azure#

This guide covers deploying NIM LLM on Microsoft Azure, including standard NIM LLM and Dynamo mode paths:

Deployment Path

Description

AKS (Azure Kubernetes Service)

Self-managed Kubernetes deployment using the NIM LLM Helm chart. Provides full control over infrastructure, including GPU node pools and persistent storage.

Microsoft Foundry

Managed deployment using Azure ML Managed Online Endpoints. Simplifies infrastructure management with built-in health checks.

Dynamo mode on AKS or Foundry

AKS DynamoGraphDeployment (DGD) and Foundry single-container paths. Refer to Dynamo Deployment.

AKS Deployment#

To deploy NIM LLM on Azure AKS, follow these steps to create an AKS cluster with GPU nodes and prepare the environment for running NIM LLM workloads. This section will walk you through prerequisites, cluster creation, and initial setup.

Prerequisites#

Install the following tools before proceeding:

Tip

If you use preview AKS features, also install the aks-preview extension.

Log in to Azure:

az login --use-device-code

You also need an NGC API key with access to NIM LLM container images and Helm charts.

Create an AKS Cluster with GPU Nodes#

Complete the following steps to create the AKS cluster, add a GPU node pool, and get the cluster credentials.

  1. Create a resource group and AKS cluster:

    export RESOURCE_GROUP="nim-resource-group"
export REGION="southcentralus"
export AKS_NAME="nim-cluster"

az group create --name $RESOURCE_GROUP --location $REGION
az aks create --resource-group $RESOURCE_GROUP --name $AKS_NAME \
  --location $REGION --generate-ssh-keys

    For detailed steps, refer to the AKS quickstart guide.

  2. Add a GPU node pool. Skip automatic GPU driver installation so the NVIDIA GPU Operator manages drivers later in this guide.

    az aks nodepool add --resource-group $RESOURCE_GROUP --cluster-name $AKS_NAME \
  --name gpupool --node-count 1 --node-vm-size Standard_ND96amsr_A100_v4 \
  --gpu-driver none --node-osdisk-size 2048 --max-pods 110

    Note

    Azure CLI 2.72.2 or later is required for --gpu-driver none. On older CLI versions, use --skip-gpu-driver-install instead (deprecated after August 2025).

    Tip

    Choose a GPU-enabled VM size that matches your model requirements. For supported options, refer to GPU workloads in AKS.

  3. Get the cluster credentials:

    az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_NAME \
  --overwrite-existing

Install the NVIDIA GPU Operator#

Install the NVIDIA GPU Operator after the AKS cluster is ready.

  1. Add the NVIDIA Helm repository:

    helm repo add nvidia https://helm.ngc.nvidia.com/nvidia --pass-credentials

  2. Update the local Helm repository index:

    helm repo update

  3. Install the GPU Operator:

    helm install --create-namespace --namespace gpu-operator \
  nvidia/gpu-operator --wait --generate-name

For more information, refer to the NVIDIA GPU Operator documentation.

Create Kubernetes Secrets#

Create the namespace and secrets that the Helm deployment uses.

  1. Set the required environment variables:

    export NGC_API_KEY="${YOUR_NGC_API_KEY}"
export NAMESPACE="nim-llm"
export IMAGE_PULL_SECRET="ngc-secret"

  2. Create the namespace:

    kubectl create namespace $NAMESPACE

  3. Create the NGC image pull secret:

    kubectl create secret docker-registry $IMAGE_PULL_SECRET \
  --namespace $NAMESPACE \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password="$NGC_API_KEY"

  4. Create the NGC API key secret:

    kubectl create secret generic ngc-api \
  --namespace $NAMESPACE \
  --from-literal=NGC_API_KEY="$NGC_API_KEY"

Optional: For gated Hugging Face models, create an additional secret.

  1. Set the Hugging Face token:

    export HF_TOKEN="${YOUR_HF_TOKEN}"

  2. Create the secret:

    kubectl create secret generic hf-token \
  --namespace $NAMESPACE \
  --from-literal=HF_TOKEN="$HF_TOKEN"

Deploy NIM LLM with Helm#

To deploy NVIDIA NIM for LLMs with Helm, follow these steps:

  1. Fetch the Helm chart from NGC:

    export NIM_LLM_CHART_VERSION="1.0.0"   # Set to your desired chart version
helm fetch https://helm.ngc.nvidia.com/nim/charts/nim-llm-${NIM_LLM_CHART_VERSION}.tgz \
  --username='$oauthtoken' --password=$NGC_API_KEY

  2. Optional: View the default chart values to understand available configuration options:

    helm show values nim-llm-${NIM_LLM_CHART_VERSION}.tgz

    Tip

    For help choosing the right model configuration for your values.yaml, refer to Model Profiles and Selection.

  3. Deploy using a custom values file:

    helm install my-nim nim-llm-${NIM_LLM_CHART_VERSION}.tgz \
  --namespace $NAMESPACE \
  -f path/to/your/custom-values.yaml

Verify the Deployment#

Complete the following steps to confirm that the service is reachable and serving inference requests.

  1. Get the service endpoint:

    kubectl -n $NAMESPACE get svc -l app.kubernetes.io/name=nim-llm

  2. Check the health endpoint (set NIM_EXTERNAL_IP from the service EXTERNAL-IP):

    export NIM_EXTERNAL_IP=$(kubectl -n $NAMESPACE get svc -l app.kubernetes.io/name=nim-llm -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')
curl -s "http://${NIM_EXTERNAL_IP}:8000/v1/health/ready"

  3. Send an inference request:

    curl -X POST "http://${NIM_EXTERNAL_IP}:8000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/llama3.1-8b-instruct",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 128
  }'

Microsoft Foundry Deployment#

To deploy NIM on Microsoft Foundry, follow these steps to provision the necessary Azure Machine Learning resources and prepare your environment for running NIM workloads. This section will walk you through prerequisites, workspace setup, and deployment of NIM as a managed online endpoint.

Prerequisites#

Before deploying NIM to Microsoft Foundry, make sure you have the following:

  • The Azure CLI and Azure ML extension:

    az extension add -n ml -y

  • An Azure Machine Learning workspace:

    az ml workspace create --name ${WORKSPACE_NAME} --resource-group ${RESOURCE_GROUP}

  • An NGC API key for pulling NIM container images and downloading model artifacts.

  • The Azure AI ML Python SDK:

    pip install azure-ai-ml azure-identity

Note

Some Azure regions have limited GPU quota. If you encounter quota errors, refer to Troubleshoot online endpoints: OutOfQuota.

Deploy with the Python SDK#

The following steps deploy a NIM container as an Azure ML managed online endpoint.

Tip

The nim_model_profile variable determines which model configuration is used. For help choosing the right profile, refer to Model Profiles and Selection.

  1. Create the ML client and endpoint:

    from azure.ai.ml import MLClient
from azure.ai.ml.entities import (
    ManagedOnlineEndpoint,
    ManagedOnlineDeployment,
    Environment,
    OnlineRequestSettings,
    ProbeSettings,
)
from azure.identity import DefaultAzureCredential

subscription_id = "<your-subscription-id>"
resource_group = "<your-resource-group>"
workspace_name = "<your-workspace-name>"
endpoint_name = "<your-endpoint-name>"
acr_image = "<your-acr-or-ngc-nim-image-uri>"
ngc_api_key = "<your-ngc-api-key>"
nim_model_profile = "<your-nim-model-profile>"
instance_type = "Standard_NC24ads_A100_v4"

client = MLClient(
    credential=DefaultAzureCredential(),
    subscription_id=subscription_id,
    resource_group_name=resource_group,
    workspace_name=workspace_name,
)

endpoint = ManagedOnlineEndpoint(
    name=endpoint_name,
    description=f"NIM deployment: {acr_image}",
    auth_mode="key",
)
client.online_endpoints.begin_create_or_update(endpoint).result()

  2. Define the environment and deployment:

    env = Environment(
    name=f"nim-env-{endpoint_name}",
    image=acr_image,
    inference_config={
        "liveness_route": {"port": 8000, "path": "/v1/health/live"},
        "readiness_route": {"port": 8000, "path": "/v1/health/ready"},
        "scoring_route": {"port": 8000, "path": "/"},
    },
)

deployment = ManagedOnlineDeployment(
    name="nim",
    endpoint_name=endpoint_name,
    environment=env,
    instance_type=instance_type,
    instance_count=1,
    environment_variables={
        "NGC_API_KEY": ngc_api_key,
        "NIM_MODEL_PROFILE": nim_model_profile,
    },
    request_settings=OnlineRequestSettings(
        request_timeout_ms=90000,
        max_concurrent_requests_per_instance=1,
    ),
    liveness_probe=ProbeSettings(
        initial_delay=600, period=30, timeout=10, failure_threshold=30
    ),
    readiness_probe=ProbeSettings(
        initial_delay=600, period=30, timeout=10, failure_threshold=30
    ),
)
client.online_deployments.begin_create_or_update(deployment).result()

  3. Route traffic to the deployment:

    endpoint = client.online_endpoints.get(endpoint_name)
endpoint.traffic = {"nim": 100}
client.online_endpoints.begin_create_or_update(endpoint).result()

  4. Retrieve the endpoint URL and key:

    endpoint_info = client.online_endpoints.get(endpoint_name)
keys = client.online_endpoints.get_keys(endpoint_name)
endpoint_url = endpoint_info.scoring_uri.rstrip("/")
api_key = keys.primary_key

Verify the Deployment#

Send an inference request to the endpoint:

import httpx

headers = {
    "Authorization": f"Bearer {api_key}",
    "Content-Type": "application/json",
}

payload = {
    "model": "meta/llama3.1-8b-instruct",  # or your deployed model name
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 128,
}

with httpx.Client(timeout=60) as http:
    response = http.post(
        f"{endpoint_url}/v1/chat/completions",
        headers=headers,
        json=payload,
    )
    response.raise_for_status()
    print(response.json())

Dynamo Deployment#

Deploy NIM with Dynamo on Microsoft Azure by using one of these paths:

Path

Guide

AKS, multi-pod DGD (Dynamo Operator)

AKS DGD

AI Foundry, single container

AI Foundry

For general Azure cluster setup (subscription, AKS creation, GPU node pools, and Foundry workspace), refer to the preceding sections on this page.

Prerequisites#

Requirement

Notes

Azure subscription

With quota for GPU VMs in your target region

Region

Choose a region where your GPU SKU is available (for example southcentralus)

Tools

az, kubectl, helm 3.x

Cluster

AKS 1.24+ with a GPU node pool; NVIDIA GPU Operator installed

Secrets

NGC_API_KEY, HF_TOKEN (required for gated Hugging Face models)

Container image

A NIM 3.0 vLLM image published for Dynamo mode

Verify GPUs are visible after GPU Operator install:

kubectl get nodes -o jsonpath='{.items[*].status.allocatable.nvidia\.com/gpu}{"\n"}'

AKS + DynamoGraphDeployment (DGD)#

Deploy the multi-pod topology with the NVIDIA Dynamo Operator and the DGD recipe published in the model’s NGC collection. The NIM-LLM Helm chart does not support DGD mode in this release, and the release does not publish a separate Helm chart for Dynamo mode.

Complete the shared prerequisites, including the AKS GPU node pool and namespace credentials. Then follow Dynamo Graph Deployment on Kubernetes. That procedure:

  • Installs the validated dynamo-platform 1.2.0 chart, which includes the required CRDs and needs no additional component flags.

  • Downloads the model-, GPU-, and workload-specific DGD recipe from the model’s NGC collection.

  • Preserves the image, profile, GPU count, replica count, and worker tuning validated with that recipe.

  • Uses /v1/health/ready for Frontend readiness.

Azure AI Foundry (managed online endpoint)#

Deploy NIM in Dynamo mode on Azure Machine Learning managed online endpoints as a single container with NIM_DYNAMO_SINGLE=1. The container runs a Dynamo frontend and one worker with file discovery and ZMQ.

This path does not use Kubernetes or the Dynamo Operator. Multi-pod DGD remains AKS-only (refer to AKS + DynamoGraphDeployment (DGD)).

For workspace creation and GPU quota requests, refer to the Azure CSP deployment guide.

Prerequisites#

Requirement

Notes

Azure ML workspace

In a region with GPU quota for your instance type

Tools

az with ml extension, Python 3.10+

Python packages

azure-ai-ml, azure-identity

Credentials

NGC_API_KEY, HF_TOKEN (for gated Hugging Face models)

Container image

A NIM 3.0 vLLM image published for Dynamo mode

RBAC

Azure ML Data Scientist or Contributor on the workspace

 
az extension add -n ml -y
pip install azure-ai-ml azure-identity
az login

Request GPU quota in the AML workspace if deploy fails with OutOfQuota. Refer to troubleshooting online endpoints.

Step 1: Environment variables#

cp deploy/azure/env.example deploy/azure/.env
# Edit .env with subscription, workspace, image, and credentials.
source deploy/azure/.env
export FOUNDRY_HEALTH_PATH=/v1/health/ready

Step 2: Deploy#

From the repository root:

python3 deploy/azure/foundry/deploy_nim_d_foundry.py deploy

The script creates (or updates):

  1. A ManagedOnlineEndpoint with key auth

  2. A ManagedOnlineDeployment with Dynamo mode environment variables

  3. 100% traffic to the deployment

Key Dynamo mode settings:

Variable

Purpose

NIM_DYNAMO_SINGLE=1

Single-container Dynamo mode

NIM_MODEL_PATH

Model URI from a supported source, such as Hugging Face or NGC

NIM_SERVED_MODEL_NAME

OpenAI API model id

HF_HOME

Hugging Face cache directory in the container

NIM_BACKEND_PORT

Internal vLLM port (default 8001)

FOUNDRY_HEALTH_PATH

Frontend management readiness route (/v1/health/ready)

The deployment helper reads FOUNDRY_HEALTH_PATH and configures both managed endpoint probes on port 8000.

First startup can take 15-45 minutes (model download + Dynamo bootstrap). Default probe initial_delay is 600 seconds (FOUNDRY_PROBE_INITIAL_DELAY).

Step 3: Smoke test#

After deploy, the script prints the scoring_uri and a command that fetches the endpoint key at run time. The key is never printed, so it does not leak into terminal history or CI logs. In another shell:

export SMOKE_BASE_URL="https://<endpoint>.<region>.inference.ml.azure.com"
export SMOKE_API_KEY=$(az ml online-endpoint get-credentials \
  -n "$ENDPOINT_NAME" -g "$RESOURCE_GROUP" -w "$WORKSPACE_NAME" \
  --query primaryKey -o tsv)
export SMOKE_MODEL="TinyLlama/TinyLlama-1.1B-Chat-v1.0"   # optional override

deploy/azure/scripts/smoke-test.sh foundry

Teardown#

source deploy/azure/.env
python3 deploy/azure/foundry/deploy_nim_d_foundry.py delete

Azure notes#

Topic

Guidance

Startup time

Use generous probe delays while NIM downloads weights and starts the Dynamo frontend and worker

GPU memory

Size INSTANCE_TYPE to your model; lower NIM_KV_CACHE_PERCENT if OOM

Auth

Endpoint uses key auth; pass Authorization: Bearer <key> on inference requests

Storage

Managed endpoints use ephemeral container disk; large models need sufficient instance disk

RBAC

Workspace read is not enough; you need deploy permissions on the AML workspace

Customize for your model#

For single-container Foundry deployments, set the model and resource controls supported by the selected NIM image and keep NIM_DYNAMO_SINGLE=1. Use /v1/health/ready for frontend readiness.

For the Kubernetes workflow, refer to Dynamo Graph Deployment on Kubernetes. Do not replace the image, model profile, GPU count, or workload tuning in the recipe downloaded from the model’s NGC collection. The published recipe and image are validated as one configuration.

Override the smoke-test model with export SMOKE_MODEL=<your-model-id> if /v1/models lists more than one id.

References#

Refer to the following Azure documentation for more information: