Upgrading NVIDIA NIM Operator#

This page outlines how to perform Kubernetes rolling upgrades on the NIM Operator container. It is possible to upgrade the NIM Operator even if you have operands (microservices) running, as upgrading the NIM Operator does not impact any running NIM or NeMo microservices custom resources (CR).

Using Helm#

Because Helm does not support automatic upgrade of existing CRDs, you can upgrade the CRDs by enabling a Helm hook or manually.

Procedure#

By default, the Helm chart enables a pre-upgrade Helm hook to update the CRDs during the Operator upgrade.

  1. Update the information about the Operator chart:

    $ helm repo update nvidia

  2. Fetch the values from the chart:

    $ helm show values nvidia/k8s-nim-operator --version=v2.0.0 > values-2.0.0.yaml

  3. Update the values file as needed.

  4. Upgrade the Operator:

    $ helm upgrade nim-operator nvidia/k8s-nim-operator -n nim-operator -f values-2.0.0.yaml

Helm hooks used with the Operator use the Operator image itself. If the Operator image itself cannot be pulled successfully, such as due to network error, the hooks fail. In this case, you need to delete the chart using the --no-hooks argument to avoid hanging the deletion due to hook failures.

Alternative: Manually Upgrading CRDs#

With this procedure, you can update all existing Operator custom resources.

  1. Apply the custom resource definitions:

    $ kubectl apply -f \
    https://raw.githubusercontent.com/NVIDIA/k8s-nim-operator/refs/tags/v2.0.0/deployments/helm/k8s-nim-operator/crds/apps.nvidia.com_nimcaches.yaml

$ kubectl apply -f \
    https://raw.githubusercontent.com/NVIDIA/k8s-nim-operator/refs/tags/v2.0.0/deployments/helm/k8s-nim-operator/crds/apps.nvidia.com_nimpipelines.yaml

$ kubectl apply -f \
    https://raw.githubusercontent.com/NVIDIA/k8s-nim-operator/refs/tags/v2.0.0/deployments/helm/k8s-nim-operator/crds/apps.nvidia.com_nimservices.yaml

  2. Update the information about the Operator chart:

    $ helm repo update nvidia

  3. Fetch the values from the chart:

    $ helm show values nvidia/k8s-nim-operator --version=v2.0.0 > values-2.0.0.yaml

  4. Update the values file as needed.

  5. Upgrade the Operator:

    $ helm upgrade nim-operator nvidia/k8s-nim-operator -n nim-operator -f values-2.0.0.yaml