Installing NVIDIA NIM Operator#

The page outlines the prerequisites and installation instructions for the NVIDIA NIM Operator.

Prerequisites#

  1. Available resources and GPU that meet the requirements for the NVIDIA NIM for LLM or NVIDIA NeMo microservices you plan to deploy. Refer to Platform Support for information on model resource requirements, supported operating systems, and Kubernetes platforms.

  2. A Kubernetes cluster with any platform-specific prerequisites configured and the cluster-admin role.

  3. An active subscription to an NVIDIA AI Enterprise product or be an NVIDIA Developer Program member. Access to the containers and models for NVIDIA NIM microservices is restricted.

  4. NVIDIA GPU Operator installed. Use the NVIDIA GPU Operator to install, configure, and manage the NVIDIA GPU driver and NVIDIA container runtime on the Kubernetes nodes. The NIM Operator requires the GPU Operator be installed on your cluster. Default installation steps are listed below. For more information or to adjust the configuration, refer to Installing the NVIDIA GPU Operator in the NVIDIA GPU Operator documentation.

    1. Add the Helm repository for NVIDIA:

      $ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \
   && helm repo update

    2. Install the GPU Operator:

      $ helm install --wait --generate-name \
   -n gpu-operator --create-namespace \
   nvidia/gpu-operator

Note

If you are installing on Red Hat Openshift, following the steps in Installing NIM Operator on Red Hat OpenShift Using the Web Console.

If you are installing on any other Kubernetes platform or Cloud Service Provider managed Kubernetes, including TKG, EKS, AKS, OKE, GKE follow the steps in Install the NIM Operator on Any Kubernetes Platform.

Install the NIM Operator on Any Kubernetes Platform#

Use the following steps to install the NVIDIA NIM Operator on your Kubernetes cluster.

  1. Create the Operator namespace:

    $ kubectl create namespace nim-operator

  2. Install the Operator:

    $ helm install nim-operator nvidia/k8s-nim-operator -n nim-operator --version=v2.0.0

  3. Optionally, confirm the controller pod is running:

    $ kubectl get pods -n nim-operator

    Example Output

    NAME                                            READY   STATUS    RESTARTS      AGE
nim-operator-k8s-nim-operator-6b546f57d5-g4zgg  1/1     Running     0           35h

Installing NIM Operator on Red Hat OpenShift Using the Web Console#

  1. In the OpenShift Container Platform web console from the side menu, navigate to Operators > OperatorHub.

  2. In Operators > OperatorHub, search for the nim-operator. For additional information see the Red Hat OpenShift Container Platform documentation.

    _images/operator-hub-search.png

  3. Select the NVIDIA NIM Operator for Kubernetes, and click Install in the subsequent screen. Its recommended that you install the latest version of the NIM Operator, but you can use the Version drop down and select a different NIM Operator version to install.

    _images/install-nim-operator.png

  4. Verify the NIM Operator is installed, navigate to Operators > Installed Operator. YOu should see the NVIDIA NIM Operator for Kubernetes listed. Click on NVIDIA NIM Operator for Kubernetes toview details about the NIM Operator.

    _images/nim-operator-details.png

Installing NIM Operator on Red Hat OpenShift using operator-sdk (for development-only)#

  1. Create the Operator namespace:

    $ oc create namespace nvidia-nim-operator

  2. Add a Docker registry secret that the Operator uses for pulling containers and models from NGC:

    $ oc create secret -n nvidia-nim-operator docker-registry ngc-secret \
    --docker-server=nvcr.io \
    --docker-username='$oauthtoken' \
    --docker-password=<ngc-api-key>

  3. Install the Operator:

    $ operator-sdk run bundle ghcr.io/nvidia/k8s-nim-operator:bundle-latest-main --namespace nvidia-nim-operator

  4. Optional: Confirm the controller pod is running:

    $ oc get pods -n nvidia-nim-operator

    Example Output

    NAME                                                              READY   STATUS      RESTARTS   AGE
ec60a4439c710b89fc2582f5384382b4241f9aee62bb3182b8d128e69d4jqfm   0/1     Completed   0          74m
ghcr-io-nvidia-k8s-nim-operator-bundle-latest-main                1/1     Running     0          75m
k8s-nim-operator-77bf775c88-bscjg                                 1/1     Running     0          74m

Next Steps#