Is this page helpful?

Helm and Kubernetes#

Deploy NIM LLM and VLM on Kubernetes with the NIM Helm chart. Use this page to fetch the chart, set the values that a typical deployment needs, install a release, and verify that the service is ready.

Prerequisites#

Before you start, complete the following prerequisites:

Provision a Kubernetes v1.26 or later cluster with GPU-capable nodes. Use a supported Kubernetes release and the latest available patch version.
Configure kubectl access to the target cluster.
Install Helm 3.0.0 or later.
Obtain an NGC API key for accessing the NIM Helm chart. You also need an NGC API key for NIM container images or model artifacts that do not support keyless access, such as PB, pre-2.0.10 FB, private, or gated assets.
Identify a storage class that supports persistent volumes for model caching.
Optional: Install the LeaderWorkerSet CRD and controller in the cluster for multi-node deployments.

Note

To provision Kubernetes with NVIDIA Cloud Native Stack (CNS), refer to Using the Ansible Playbooks.

Fetch the Helm Chart#

To fetch and extract the Helm chart, complete the following steps:

Open the NGC Catalog and select the nim-llm Helm chart to pick a version. In most cases, you should select the latest version.

Fetch and extract the chart:

export HELM_CHART_VERSION="<version_number>"
helm fetch "https://helm.ngc.nvidia.com/nim/charts/nim-llm-${HELM_CHART_VERSION}.tgz" \
  --username='$oauthtoken' \
  --password="${NGC_API_KEY}"
tar -xzf "nim-llm-${HELM_CHART_VERSION}.tgz"

Configure Helm Values#

To inspect defaults and set the values that a NIM deployment needs, complete the following steps:

Inspect chart documentation and defaults:

helm show readme nim-llm/
helm show values nim-llm/

Set the following Helm values as needed for your cluster and model:
- image.repository: NIM container image to deploy.
- image.tag: NIM container image tag.
- model.ngcAPISecret and imagePullSecrets: optional credentials used for images and model artifacts that require an NGC API key.
- persistence: storage settings for model cache.
- resources: GPU limits based on model requirements.
- customArgs: backend arguments passed directly to the NIM container.
- env: optional advanced runtime configuration.
- statefulSet.enabled / deployment.strategy: use Deployment mode with an update strategy such as Recreate when spare GPU capacity is limited during upgrades. Refer to Set the Deployment Update Strategy.
Optional: Pass backend arguments with customArgs. This preserves any NIM_PASSTHROUGH_ARGS value provided by the image and applies the custom arguments at the higher CLI priority. Specify each option and value as a separate list item. The following example uses vLLM arguments:
```
customArgs:
  - "--enable-prefix-caching"
  - "--max-num-batched-tokens"
  - "8192"
```
When multiNode.enabled=true, the chart passes customArgs to the leader container, which launches the backend. Worker containers do not receive them.

Argument names and supported values must match the backend version in the image. If NIM_STRICT_ARG_PROCESSING=true, the NIM exits when a custom argument conflicts with an image-provided passthrough value.

Do not set NIM_PASSTHROUGH_ARGS under env to only your custom arguments. Kubernetes replaces any value supplied by the image instead of appending to it. If an environment variable is the only available interface, supply the complete combined value as described in Advanced Configuration.

Optional: Review cache and temporary directories. Several environment variables are derived from model.nimCache (default: /model-store) at deploy time:

Environment Variable	Derived Value	Purpose
`NIM_CACHE_PATH`	`<nimCache>`	Primary model cache
`HF_HOME`	`<nimCache>/huggingface/hub`	Hugging Face cache
`OUTLINES_CACHE_DIR`	`<nimCache>/outlines`	Outlines grammar cache for structured output

These variables are set automatically in both single-node and multi-node deployments. If you change model.nimCache, confirm that the underlying volume mount is writable and has sufficient space for all cache subdirectories.

To override OUTLINES_CACHE_DIR independently, add it to the env section in values.yaml:

env:
  - name: OUTLINES_CACHE_DIR
    value: /custom/path/outlines

Deploy a Minimal Helm Release#

To deploy the minimal Helm example for an eligible public-catalog NIM image and tag that supports keyless access, complete the following steps:

Create values.yaml with a minimal configuration:

cat <<'EOF' > values.yaml
image:
  repository: <NIM_LLM_MODEL_SPECIFIC_IMAGE>
  tag: "2.0.13"
model:
  ngcAPISecret: ""
persistence:
  enabled: true
  storageClass: "nfs-client"
  accessMode: ReadWriteMany
  size: 50Gi
resources:
  limits:
    nvidia.com/gpu: 1
imagePullSecrets: []
EOF

Note

Set persistence.storageClass to a StorageClass that is available in your Kubernetes cluster.

Tip

Adjust persistence.size based on your model size and expected cache usage.

Install the release:

helm install my-nim nim-llm/ -f values.yaml

These values are intentionally minimal and work as a starting point in most clusters.

Add NGC Credentials When Required#

If your image, tag, or model artifact requires an NGC API key, create the secrets before installing the chart. To add NGC credentials, complete the following steps:

Export your NGC API key:
```
export NGC_API_KEY=<your_ngc_api_key>
```

Create the image pull secret:

kubectl create secret docker-registry ngc-secret \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password="${NGC_API_KEY}"

Create the NGC API key secret:

kubectl create secret generic nvidia-nim-secrets \
  --from-literal=NGC_API_KEY="${NGC_API_KEY}"

Update the credential values in values.yaml:

model:
  ngcAPISecret: "nvidia-nim-secrets"
imagePullSecrets:
  - name: "ngc-secret"

Enable LoRA with Helm#

Optional: To enable LoRA adapters with Helm, complete the following steps:

Create a dedicated PVC for the LoRA adapters:

cat <<'EOF' > nvidia-nim-lora-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nvidia-nim-lora-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: nfs-client
  resources:
    requests:
      storage: 10Gi
EOF

kubectl apply -f nvidia-nim-lora-pvc.yaml

Add LoRA adapters to the PVC under /loras. For each adapter, create one directory that contains the adapter artifacts:
```
/loras/
  adapter_name/
    adapter_config.json
    adapter_model.safetensors   # or adapter_model.bin
```
Note

NIM loads adapters from NIM_PEFT_SOURCE (/loras in this example). If the PVC is empty, no LoRA adapters are available at runtime.

Update values.yaml:

env:
  - name: NIM_PEFT_SOURCE
    value: /loras
extraVolumes:
  lora-adapter:
    persistentVolumeClaim:
      claimName: nvidia-nim-lora-pvc
extraVolumeMounts:
  lora-adapter:
    mountPath: /loras

Apply the updated values:

helm upgrade my-nim nim-llm/ -f values.yaml

For detailed LoRA configuration and runtime behavior, refer to Fine-Tuning with LoRA.

Configure Container Security Context#

The containerSecurityContext Helm value sets the Kubernetes container-level security context for the NIM container. It is applied in both single-node and multi-node (LeaderWorkerSet) deployments.

Unlike podSecurityContext, the container-level security context supports the capabilities field, which is required for certain multi-node configurations.

LeaderWorkerSet Requirement#

When you set multiNode.enabled=true, the chart creates LeaderWorkerSet resources. Install the LeaderWorkerSet CRD and controller before installing the NIM Helm chart. LeaderWorkerSet requires Kubernetes v1.26 or later.

NVL72 Multi-Node Deployments#

GB200 and GB300 NVL72 systems use IMEX (Inter-Memory Exchange) channels for cross-node NVLink communication. The containers must have the following Linux capabilities:

containerSecurityContext:
  capabilities:
    add:
      - SYS_PTRACE
      - IPC_LOCK

SYS_PTRACE: Required for CUDA IPC and IMEX channel operations across GPUs.
IPC_LOCK: Required for pinning memory used by GPUDirect RDMA.

To add these capabilities with Helm, complete one of the following steps:

Pass the capabilities on the Helm command line:

helm install my-nim nim-llm/ -f values.yaml \
  --set containerSecurityContext.capabilities.add[0]=SYS_PTRACE \
  --set containerSecurityContext.capabilities.add[1]=IPC_LOCK

Add them to your values.yaml instead:

containerSecurityContext:
  capabilities:
    add:
      - SYS_PTRACE
      - IPC_LOCK

Set the Deployment Update Strategy#

By default the chart deploys a StatefulSet (statefulSet.enabled: true). When you set statefulSet.enabled: false, the chart creates a Deployment instead. In that mode you can set deployment.strategy to control Kubernetes Deployment update strategy.

To use type: Recreate when the cluster cannot spare an extra GPU pod during upgrades (the default RollingUpdate strategy needs room for a new pod before the old one terminates), add the following values:

statefulSet:
  enabled: false
deployment:
  strategy:
    type: Recreate

This value applies only in Deployment mode. It is ignored when statefulSet.enabled is true.

Assign a Pod Priority Class#

Use the priorityClassName Helm value to assign a Kubernetes PriorityClass to NIM pods. Without a priority class, pods are treated as lowest priority in the cluster and can be preempted by other workloads.

This is applied to single-node deployments and multi-node (LeaderWorkerSet) leader and worker pods.

To assign a PriorityClass, complete the following steps:

Create a PriorityClass if it does not already exist in the cluster:

kubectl apply -f - <<'EOF'
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "High priority class for NIM inference pods."
EOF

Set priorityClassName in your values file:
```
priorityClassName: "high-priority"
```

Verify the Deployment#

To verify that the pods and service are ready, complete the following steps:

Check that the pods are running:

kubectl get pods -l app.kubernetes.io/instance=my-nim

Check that the service is available:

kubectl get svc -l app.kubernetes.io/instance=my-nim

Port-forward the service for local testing:

kubectl port-forward svc/my-nim-nim-llm 8000:8000

Call the readiness endpoint to confirm that the service is ready:
```
curl -sS http://127.0.0.1:8000/v1/health/ready
```

A healthy deployment returns an HTTP 200 response from the readiness endpoint.

Next Steps#

After the Helm release is running, continue with the following topics:

NIM Operator Deployment to manage NIM with Kubernetes custom resources
KServe to deploy NIM as a KServe InferenceService
Fine-Tuning with LoRA for LoRA runtime behavior
Environment Variables for the full configuration reference