Installing NVIDIA NIM Operator

Prerequisites

• A Kubernetes cluster and the cluster-admin role. Refer to Platform Support for information about supported operating systems and Kubernetes platforms.

• NVIDIA A100 80 GB, H100, or L40S GPUs on one or more nodes. Refer to Platform Support for information about models and required GPU model and GPU count. For large models that exceed the memory capacity of one GPU, you need to add more GPUs. When you deploy a pipeline, you can specify more than one GPU for a workload.

• An active subscription to an NVIDIA AI Enterprise product or be an NVIDIA Developer Program member. Access to the containers and models for NVIDIA NIM microservices is restricted.

Installing GPU Operator

Use the NVIDIA GPU Operator to install, configure, and manage the NVIDIA GPU driver and NVIDIA container runtime on the Kubernetes nodes.

1. Add the Helm repository for NVIDIA:

   $ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \
      && helm repo update
   

2. Install the Operator:

   $ helm install --wait --generate-name \
      -n gpu-operator --create-namespace \
      nvidia/gpu-operator
   

For more information or to adjust the configuration, refer to Installing the NVIDIA GPU Operator in the NVIDIA GPU Operator documentation.

Install NIM Operator

1. Create the Operator namespace:

   $ kubectl create namespace nim-operator
   

2. Install the Operator:

   $ helm install nim-operator nvidia/k8s-nim-operator -n nim-operator --version=v1.0.1
   

3. Optional: Confirm the controller pod is running:

   $ kubectl get pods -n nim-operator
   

   Example Output

   NAME                                            READY   STATUS    RESTARTS      AGE
   nim-operator-k8s-nim-operator-6b546f57d5-g4zgg  2/2     Running     0           35h
   

Next Steps

• Refer to Caching Models to download and cache inference and embedding models.