Install NeMo Platform Helm Chart

Complete Prerequisites before installing the chart.

Set install variables

Use a pinned chart version for every install and upgrade. For a stable release, use the chart version from the release notes or NGC Catalog. For nightly GHCR charts, the release workflow uses the latest stable SemVer release tag and appends a nightly timestamp, falling back to k8s/helm/Chart.yaml if no release tag is available.

export NMP_NAMESPACE=nemo-platform
export NMP_HELM_RELEASE=nemo-platform
export NMP_HELM_CHART_VERSION=<chart-version>
export NGC_API_KEY=<your-ngc-api-key>

Create the namespace if you have not already created it:

kubectl create namespace "$NMP_NAMESPACE"

Fetch the chart

helm repo add nemo-platform https://helm.ngc.nvidia.com/nvidia/nemo-platform \
  --username '$oauthtoken' \
  --password "$NGC_API_KEY"
helm repo update nemo-platform
helm show values nemo-platform/nemo-platform \
  --version "$NMP_HELM_CHART_VERSION" > values.yaml

Configure values

At minimum, make sure the chart can use the image pull secret and NGC API secret from Prerequisites:

existingSecret: ngc-api
imagePullSecrets:
  - name: nvcrimagepullsecret

Review the NeMo Platform Helm Chart reference before installing. The most common cluster-specific settings are:

Local single-node clusters

For minikube or kind, the default StorageClass often does not satisfy a ReadWriteMany claim. For a single-node QA smoke test, create a static hostPath PV and a matching StorageClass:

cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nemo-local-rwx
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nemo-platform-local-rwx
spec:
  capacity:
    storage: 200Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Delete
  storageClassName: nemo-local-rwx
  hostPath:
    path: /var/lib/nemo-platform
EOF

Add the StorageClass to values.yaml:

core:
  storage:
    storageClass: nemo-local-rwx
    accessModes:
      - ReadWriteMany
    size: 200Gi

For local API access without ingress, use port-forwarding after the release is ready:

kubectl -n "$NMP_NAMESPACE" port-forward svc/nemo-platform-api 8080:8080

EKS example

For EKS, use a ReadWriteMany-capable StorageClass for jobs storage. The example below assumes an EFS CSI StorageClass named efs-sc and an EBS CSI StorageClass named gp3:

core:
  storage:
    storageClass: efs-sc
    accessModes:
      - ReadWriteMany
    size: 200Gi
postgresql:
  persistence:
    storageClass: gp3
    size: 20Gi

If you store Files service data in S3, configure the file storage backend and make AWS credentials available through IRSA or another boto3-supported credential source:

platformConfig:
  files:
    default_storage_config:
      type: s3
      bucket: my-nemo-platform-bucket
      region: us-east-1
      use_sdk_auth: true

For multi-node GPU jobs on EKS, also configure EFA networking as described in Multinode Networking.

Install Volcano

Install the Volcano scheduler before installing the chart. This is required for customization jobs that leverage multiple nodes.

kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.9.0/installer/volcano-development.yaml
kubectl wait --for=condition=complete job/volcano-admission-init -n volcano-system --timeout=120s
kubectl rollout status deployment/volcano-admission -n volcano-system

After applying Volcano, wait for the admission webhook to finish initializing before proceeding. The webhook registers immediately with failurePolicy: Fail, but TLS certificate generation runs asynchronously. If you proceed before the webhook is ready, pod creation can fail with certificate errors.

Install the chart

Install from the NGC Helm repository:

helm upgrade --install "$NMP_HELM_RELEASE" nemo-platform/nemo-platform \
  --version "$NMP_HELM_CHART_VERSION" \
  --namespace "$NMP_NAMESPACE" \
  --create-namespace \
  -f values.yaml \
  --wait \
  --timeout 20m

To install a nightly chart from GHCR, use the OCI reference:

helm upgrade --install "$NMP_HELM_RELEASE" "$NMP_HELM_CHART_REF" \
  --version "$NMP_HELM_CHART_VERSION" \
  --namespace "$NMP_NAMESPACE" \
  --create-namespace \
  -f values.yaml \
  --wait \
  --timeout 20m

If you pulled the chart archive directly, install from the archive:

helm upgrade --install "$NMP_HELM_RELEASE" "./nemo-platform-${NMP_HELM_CHART_VERSION}.tgz" \
  --namespace "$NMP_NAMESPACE" \
  --create-namespace \
  -f values.yaml \
  --wait \
  --timeout 20m

The installation process can take approximately 10 minutes for image downloads, container startup, and service readiness. Pods might appear in pending or restarting states while dependencies initialize.

Verify the deployment

Check the release and pod status:

helm status "$NMP_HELM_RELEASE" -n "$NMP_NAMESPACE"
kubectl get pods -n "$NMP_NAMESPACE"

Wait for the API deployment to become available:

kubectl rollout status deployment/nemo-platform-api -n "$NMP_NAMESPACE" --timeout=10m

For a local smoke test, port-forward the API service and call the readiness endpoint:

kubectl -n "$NMP_NAMESPACE" port-forward svc/nemo-platform-api 8080:8080 &
NMP_PORT_FORWARD_PID=$!
trap 'kill "$NMP_PORT_FORWARD_PID" 2>/dev/null || true' EXIT
for attempt in $(seq 1 30); do
  if curl -sf http://localhost:8080/health/ready; then
    break
  fi
  if ! kill -0 "$NMP_PORT_FORWARD_PID" 2>/dev/null; then
    echo "Port-forward process exited before the API became ready." >&2
    wait "$NMP_PORT_FORWARD_PID"
    exit 1
  fi
  if [ "$attempt" -eq 30 ]; then
    echo "API did not become ready at http://localhost:8080/health/ready." >&2
    exit 1
  fi
  sleep 2
done

Leave the shell open while running CLI commands through the forwarded port. The trap stops the port-forward when the shell exits; to stop it earlier, run kill "$NMP_PORT_FORWARD_PID".

Configure the CLI to use the deployed API:

nemo config set --base-url http://localhost:8080
nemo workspaces list

Upgrade the chart

Update NMP_HELM_CHART_VERSION, review the release notes for required values changes, and run:

helm repo update nemo-platform
helm upgrade "$NMP_HELM_RELEASE" nemo-platform/nemo-platform \
  --version "$NMP_HELM_CHART_VERSION" \
  --namespace "$NMP_NAMESPACE" \
  -f values.yaml \
  --wait \
  --timeout 20m

For a release installed from GHCR with NMP_HELM_CHART_REF, use the Helm OCI reference:

helm upgrade "$NMP_HELM_RELEASE" "$NMP_HELM_CHART_REF" \
  --version "$NMP_HELM_CHART_VERSION" \
  --namespace "$NMP_NAMESPACE" \
  -f values.yaml \
  --wait \
  --timeout 20m

Uninstall the platform

To uninstall the Helm release:

helm uninstall "$NMP_HELM_RELEASE" -n "$NMP_NAMESPACE"

• PVCs are preserved to prevent accidental data loss. Delete them manually if no longer needed.
• CRDs are not removed by Helm design (upstream issue) to avoid destroying custom resources across the cluster.
• Completed jobs, secrets, and the namespace may also remain.

If you need a complete teardown for CI/CD pipelines or reinstalling in the same namespace, delete the namespace after helm uninstall and inventory CRDs before deleting any cluster-wide API definitions:

kubectl delete namespace "$NMP_NAMESPACE"
kubectl get crd -o custom-columns=NAME:.metadata.name,API_GROUP:.spec.group

Troubleshooting

Volcano admission webhook blocks pod creation

Symptom: Pod creation fails cluster-wide with an error like:

Internal error occurred: failed calling webhook "mutatepod.volcano.sh":
failed to call webhook: Post "https://volcano-admission-service.volcano-system.svc:443/pods/mutate?timeout=10s":
tls: failed to verify certificate: x509: certificate signed by unknown authority

Cause: The Volcano MutatingWebhookConfiguration registers with failurePolicy: Fail before the volcano-admission-init job finishes generating TLS certificates. This affects all namespaces, not just Volcano workloads.

Fix: Wait for the webhook to become functional, then restart the admission deployment to force certificate regeneration:

kubectl rollout restart deployment/volcano-admission -n volcano-system
kubectl rollout status deployment/volcano-admission -n volcano-system

Verify the webhook is accepting requests before retrying your Helm install:

until kubectl run volcano-webhook-test --image=busybox --restart=Never --dry-run=server -o yaml 2>/dev/null; do
  echo "Volcano webhook not ready yet, waiting..."
  sleep 5
done
kubectl delete pod volcano-webhook-test --ignore-not-found=true