Is this page helpful?

Helm and Kubernetes#

This page describes how to deploy NVIDIA NIM for Large Language Models (LLM) and Vision Language Models (VLM) on Kubernetes using the NIM Helm chart.

Prerequisites#

Before deploying NIM with Helm, make sure you have the following:

  • A running Kubernetes v1.26 or later cluster with GPU-capable nodes. Use a supported Kubernetes release and the latest available patch version.

  • Configured kubectl access to the target cluster

  • Helm 3.0.0 or later

  • An NGC API key for accessing the NIM Helm chart. You also need an NGC API key for NIM container images or model artifacts that do not support keyless access, such as PB, pre-2.0.10, private, or gated assets.

  • A storage class that supports persistent volumes for model caching

  • For multi-node deployments, the LeaderWorkerSet CRD and controller must be installed in the cluster

Note

To provision Kubernetes with NVIDIA Cloud Native Stack (CNS), refer to Using the Ansible Playbooks.

Fetch and extract the Helm chart before deployment. Go to the NGC Catalog and select the nim-llm Helm chart to pick a version. In most cases, you should select the latest version.

export HELM_CHART_VERSION="<version_number>"
helm fetch "https://helm.ngc.nvidia.com/nim/charts/nim-llm-${HELM_CHART_VERSION}.tgz" \
  --username='$oauthtoken' \
  --password="${NGC_API_KEY}"
tar -xzf "nim-llm-${HELM_CHART_VERSION}.tgz"

Configure Helm#

The following Helm values are the most important settings for a NIM deployment:

  • image.repository: NIM container image to deploy.

  • image.tag: NIM container image tag.

  • model.ngcAPISecret and imagePullSecrets: optional credentials used for images and model artifacts that require an NGC API key.

  • persistence: storage settings for model cache.

  • resources: GPU limits based on model requirements.

  • customArgs: backend arguments passed directly to the NIM container.

  • env: optional advanced runtime configuration.

  • statefulSet.enabled / deployment.strategy: use Deployment mode with an update strategy such as Recreate when spare GPU capacity is limited during upgrades. See Deployment Update Strategy.

Use the following commands to inspect chart documentation and defaults:

helm show readme nim-llm/
helm show values nim-llm/

Pass Backend Arguments#

Use customArgs to pass backend arguments to the NIM container. This preserves any NIM_PASSTHROUGH_ARGS value provided by the image and applies the custom arguments at the higher CLI priority. Specify each option and value as a separate list item. The following example uses vLLM arguments:

customArgs:
  - "--enable-prefix-caching"
  - "--max-num-batched-tokens"
  - "8192"

When multiNode.enabled=true, the chart passes customArgs to the leader container, which launches the backend. Worker containers do not receive them.

Argument names and supported values must match the backend version in the image. If NIM_STRICT_ARG_PROCESSING=true, the NIM exits when a custom argument conflicts with an image-provided passthrough value.

Do not set NIM_PASSTHROUGH_ARGS under env to only your custom arguments. Kubernetes replaces any value supplied by the image instead of appending to it. If an environment variable is the only available interface, supply the complete combined value as described in Advanced Configuration.

Cache and Temporary Directories#

Several environment variables are derived from model.nimCache (default: /model-store) at deploy time:

Environment Variable

Derived Value

Purpose

NIM_CACHE_PATH

<nimCache>

Primary model cache

HF_HOME

<nimCache>/huggingface/hub

Hugging Face cache

OUTLINES_CACHE_DIR

<nimCache>/outlines

Outlines grammar cache for structured output

These variables are set automatically in both single-node and multi-node deployments. If you change model.nimCache, ensure that the underlying volume mount is writable and has sufficient space for all cache subdirectories.

To override OUTLINES_CACHE_DIR independently, add it to the env section in values.yaml:

env:
  - name: OUTLINES_CACHE_DIR
    value: /custom/path/outlines

Minimal Example#

Complete the following steps to deploy the minimal Helm example for an eligible public-catalog NIM image and tag that supports keyless access:

  1. Create values.yaml with a minimal configuration:

    cat <<'EOF' > values.yaml
image:
  repository: <NIM_LLM_MODEL_SPECIFIC_IMAGE>
  tag: "2.0.12"
model:
  ngcAPISecret: ""
persistence:
  enabled: true
  storageClass: "nfs-client"
  accessMode: ReadWriteMany
  size: 50Gi
resources:
  limits:
    nvidia.com/gpu: 1
imagePullSecrets: []
EOF

    Note

    Set persistence.storageClass to a StorageClass that is available in your Kubernetes cluster.

    Tip

    Adjust persistence.size based on your model size and expected cache usage.

  2. Install the release:

    helm install my-nim nim-llm/ -f values.yaml

These values are intentionally minimal and work as a starting point in most clusters.

Add NGC Credentials When Required#

If your image, tag, or model artifact requires an NGC API key, create the secrets before installing the chart:

export NGC_API_KEY=<your_ngc_api_key>

Create the image pull secret:

kubectl create secret docker-registry ngc-secret \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password="${NGC_API_KEY}"

Create the NGC API key secret:

kubectl create secret generic nvidia-nim-secrets \
  --from-literal=NGC_API_KEY="${NGC_API_KEY}"

Update the credential values in values.yaml:

model:
  ngcAPISecret: "nvidia-nim-secrets"
imagePullSecrets:
  - name: "ngc-secret"

Enable LoRA with Helm#

Optional: Complete the following steps to enable LoRA adapters with Helm.

  1. Create a dedicated PVC for the LoRA adapters:

    cat <<'EOF' > nvidia-nim-lora-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nvidia-nim-lora-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: nfs-client
  resources:
    requests:
      storage: 10Gi
EOF

kubectl apply -f nvidia-nim-lora-pvc.yaml

  2. Add LoRA adapters to the PVC under /loras. For each adapter, create one directory that contains the adapter artifacts:

    /loras/
  adapter_name/
    adapter_config.json
    adapter_model.safetensors   # or adapter_model.bin

    Note

    NIM loads adapters from NIM_PEFT_SOURCE (/loras in this example). If the PVC is empty, no LoRA adapters are available at runtime.

  3. Update values.yaml:

    env:
  - name: NIM_PEFT_SOURCE
    value: /loras
extraVolumes:
  lora-adapter:
    persistentVolumeClaim:
      claimName: nvidia-nim-lora-pvc
extraVolumeMounts:
  lora-adapter:
    mountPath: /loras

  4. Apply the updated values:

    helm upgrade my-nim nim-llm/ -f values.yaml

For detailed LoRA configuration and runtime behavior, refer to Fine-Tuning with LoRA.

Container Security Context#

The containerSecurityContext Helm value sets the Kubernetes container-level security context for the NIM container. It is applied in both single-node and multi-node (LeaderWorkerSet) deployments.

Unlike podSecurityContext, the container-level security context supports the capabilities field, which is required for certain multi-node configurations.

LeaderWorkerSet Requirement#

When you set multiNode.enabled=true, the chart creates LeaderWorkerSet resources. Install the LeaderWorkerSet CRD and controller before installing the NIM Helm chart. LeaderWorkerSet requires Kubernetes v1.26 or later.

NVL72 Multi-Node Deployments#

GB200 and GB300 NVL72 systems use IMEX (Inter-Memory Exchange) channels for cross-node NVLink communication. The containers must have the following Linux capabilities:

containerSecurityContext:
  capabilities:
    add:
      - SYS_PTRACE
      - IPC_LOCK

  • SYS_PTRACE — Required for CUDA IPC and IMEX channel operations across GPUs.

  • IPC_LOCK — Required for pinning memory used by GPUDirect RDMA.

To set these with Helm:

helm install my-nim nim-llm/ -f values.yaml \
  --set containerSecurityContext.capabilities.add[0]=SYS_PTRACE \
  --set containerSecurityContext.capabilities.add[1]=IPC_LOCK

Or add them to your values.yaml:

containerSecurityContext:
  capabilities:
    add:
      - SYS_PTRACE
      - IPC_LOCK

Deployment Update Strategy#

By default the chart deploys a StatefulSet (statefulSet.enabled: true). When you set statefulSet.enabled: false, the chart creates a Deployment instead. In that mode you can set deployment.strategy to control Kubernetes Deployment update strategy.

Use type: Recreate when the cluster cannot spare an extra GPU pod during upgrades (the default RollingUpdate strategy needs room for a new pod before the old one terminates):

statefulSet:
  enabled: false
deployment:
  strategy:
    type: Recreate

This value applies only in Deployment mode. It is ignored when statefulSet.enabled is true.

Pod Priority Class#

Use the priorityClassName Helm value to assign a Kubernetes PriorityClass to NIM pods. Without a priority class, pods are treated as lowest priority in the cluster and may be preempted by other workloads.

This is applied to single-node deployments and multi-node (LeaderWorkerSet) leader and worker pods.

priorityClassName: "high-priority"

Note

The PriorityClass resource must already exist in your cluster before referencing it.

To create a PriorityClass:

kubectl apply -f - <<'EOF'
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "High priority class for NIM inference pods."
EOF

Verify Deployment#

Verify that the pods and service are ready, and then test inference with port forwarding.

  1. Check that the pods are running:

    kubectl get pods -l app.kubernetes.io/instance=my-nim

  2. Check that the service is available:

    kubectl get svc -l app.kubernetes.io/instance=my-nim

  3. Port-forward the service for local testing:

    kubectl port-forward svc/my-nim-nim-llm 8000:8000

  4. Call the readiness endpoint to confirm that the service is ready:

    curl -sS http://127.0.0.1:8000/v1/health/ready

A healthy deployment returns an HTTP 200 response from the readiness endpoint.