Uninstall NeMo Microservices Helm Chart

To properly uninstall the NeMo microservices, follow these steps:

1. Delete all custom resource definitions (CRDs) and custom resources (CRs) of the following NeMo resource types: nemotrainingjobs, nemoentityhandlers, nemotrainingworkloads.

   Important

   This must be done while the NeMo Operator microservice is still running, before helm uninstall.

   # Step 1: Get all CRDs with 'nemo' prefix
   NEMO_CRDS="$(kubectl get crds -o name | grep 'nvidia.com' | grep 'nemo' | cut -d'/' -f2)"
   
   if [ -z "${NEMO_CRDS}" ]; then
     echo "Nemo CRDs not found"
     exit 1
   fi
   
   # Step 2: Delete all custom resource instances for each CRD
   for CRD in ${NEMO_CRDS}; do
     # Delete all instances across all namespaces
   
     echo "Deleting all instances of ${CRD}..."
     kubectl get "${CRD}" --all-namespaces -o name | xargs -r kubectl delete --wait=false
   done
   
   # Step 3: Wait for NeMo Operator to handle finalizer removal
   echo "Waiting for NeMo Operator to handle finalizer removal (30 seconds)..."
   sleep 30
   
   # Step 4: Verify all Custom Resources have been removed
   for CRD in ${NEMO_CRDS}; do
     # Check CRs
   
     echo "Checking instances of ${CRD}"
     kubectl get "${CRD}" --all-namespaces -o name
   done
   
   # Step 5: Delete the CRDs themselves
   echo "Deleting CRDs..."
   
   for CRD in ${NEMO_CRDS}; do
     # Delete all CRD's
   
     echo "Deleting CRD: ${CRD}..."
     kubectl delete crd "${CRD}"
   done
   

2. Run helm uninstall.

   helm uninstall nemo