Install NeMo Platform Helm Chart

Complete Prerequisites before installing the chart.

Set install variables

Use a pinned chart version for every install and upgrade. For a stable release, use the chart version from the release notes or NGC Catalog. For nightly GHCR charts, the release workflow uses the latest stable SemVer release tag and appends a nightly timestamp, falling back to k8s/helm/Chart.yaml if no release tag is available.

1
export NMP_NAMESPACE=nemo-platform
2
export NMP_HELM_RELEASE=nemo-platform
3
export NMP_HELM_CHART_VERSION=<chart-version>
4
export NGC_API_KEY=<your-ngc-api-key>

Create the namespace if you have not already created it:

1
kubectl create namespace "$NMP_NAMESPACE"

Fetch the chart

1
helm repo add nemo-platform https://helm.ngc.nvidia.com/nvidia/nemo-platform \
2
  --username '$oauthtoken' \
3
  --password "$NGC_API_KEY"
4
5
helm repo update nemo-platform
6
helm show values nemo-platform/nemo-platform \
7
  --version "$NMP_HELM_CHART_VERSION" > values.yaml

Configure values

At minimum, make sure the chart can use the image pull secret and NGC API secret from Prerequisites:

1
existingSecret: ngc-api
2
3
imagePullSecrets:
4
  - name: nvcrimagepullsecret

Review the NeMo Platform Helm Chart reference before installing. The most common cluster-specific settings are:

Local single-node clusters

For minikube or kind, the default StorageClass often does not satisfy a ReadWriteMany claim. For a single-node QA smoke test, create a static hostPath PV and a matching StorageClass:

1
cat <<EOF | kubectl apply -f -
2
apiVersion: storage.k8s.io/v1
3
kind: StorageClass
4
metadata:
5
  name: nemo-local-rwx
6
provisioner: kubernetes.io/no-provisioner
7
volumeBindingMode: WaitForFirstConsumer
8
---
9
apiVersion: v1
10
kind: PersistentVolume
11
metadata:
12
  name: nemo-platform-local-rwx
13
spec:
14
  capacity:
15
    storage: 200Gi
16
  accessModes:
17
    - ReadWriteMany
18
  persistentVolumeReclaimPolicy: Delete
19
  storageClassName: nemo-local-rwx
20
  hostPath:
21
    path: /var/lib/nemo-platform
22
EOF

Add the StorageClass to values.yaml:

1
core:
2
  storage:
3
    storageClass: nemo-local-rwx
4
    accessModes:
5
      - ReadWriteMany
6
    size: 200Gi

For local API access without ingress, use port-forwarding after the release is ready:

1
kubectl -n "$NMP_NAMESPACE" port-forward svc/nemo-platform-api 8080:8080

EKS example

For EKS, use a ReadWriteMany-capable StorageClass for jobs storage. The example below assumes an EFS CSI StorageClass named efs-sc and an EBS CSI StorageClass named gp3:

1
core:
2
  storage:
3
    storageClass: efs-sc
4
    accessModes:
5
      - ReadWriteMany
6
    size: 200Gi
7
8
postgresql:
9
  persistence:
10
    storageClass: gp3
11
    size: 20Gi

If you store Files service data in S3, configure the file storage backend and make AWS credentials available through IRSA or another boto3-supported credential source:

1
platformConfig:
2
  files:
3
    default_storage_config:
4
      type: s3
5
      bucket: my-nemo-platform-bucket
6
      region: us-east-1
7
      use_sdk_auth: true

For multi-node GPU jobs on EKS, also configure EFA networking as described in Multinode Networking.

Install Volcano

Install the Volcano scheduler before installing the chart. This is required for customization jobs that leverage multiple nodes.

1
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.9.0/installer/volcano-development.yaml
2
kubectl wait --for=condition=complete job/volcano-admission-init -n volcano-system --timeout=120s
3
kubectl rollout status deployment/volcano-admission -n volcano-system

After applying Volcano, wait for the admission webhook to finish initializing before proceeding. The webhook registers immediately with failurePolicy: Fail, but TLS certificate generation runs asynchronously. If you proceed before the webhook is ready, pod creation can fail with certificate errors.

Install the chart

Install from the NGC Helm repository:

1
helm upgrade --install "$NMP_HELM_RELEASE" nemo-platform/nemo-platform \
2
  --version "$NMP_HELM_CHART_VERSION" \
3
  --namespace "$NMP_NAMESPACE" \
4
  --create-namespace \
5
  -f values.yaml \
6
  --wait \
7
  --timeout 20m

To install a nightly chart from GHCR, use the OCI reference:

1
helm upgrade --install "$NMP_HELM_RELEASE" "$NMP_HELM_CHART_REF" \
2
  --version "$NMP_HELM_CHART_VERSION" \
3
  --namespace "$NMP_NAMESPACE" \
4
  --create-namespace \
5
  -f values.yaml \
6
  --wait \
7
  --timeout 20m

If you pulled the chart archive directly, install from the archive:

1
helm upgrade --install "$NMP_HELM_RELEASE" "./nemo-platform-${NMP_HELM_CHART_VERSION}.tgz" \
2
  --namespace "$NMP_NAMESPACE" \
3
  --create-namespace \
4
  -f values.yaml \
5
  --wait \
6
  --timeout 20m

The installation process can take approximately 10 minutes for image downloads, container startup, and service readiness. Pods might appear in pending or restarting states while dependencies initialize.

Verify the deployment

Check the release and pod status:

1
helm status "$NMP_HELM_RELEASE" -n "$NMP_NAMESPACE"
2
kubectl get pods -n "$NMP_NAMESPACE"

Wait for the API deployment to become available:

1
kubectl rollout status deployment/nemo-platform-api -n "$NMP_NAMESPACE" --timeout=10m

For a local smoke test, port-forward the API service and call the readiness endpoint:

1
kubectl -n "$NMP_NAMESPACE" port-forward svc/nemo-platform-api 8080:8080 &
2
NMP_PORT_FORWARD_PID=$!
3
trap 'kill "$NMP_PORT_FORWARD_PID" 2>/dev/null || true' EXIT
4
5
for attempt in $(seq 1 30); do
6
  if curl -sf http://localhost:8080/health/ready; then
7
    break
8
  fi
9
10
  if ! kill -0 "$NMP_PORT_FORWARD_PID" 2>/dev/null; then
11
    echo "Port-forward process exited before the API became ready." >&2
12
    wait "$NMP_PORT_FORWARD_PID"
13
    exit 1
14
  fi
15
16
  if [ "$attempt" -eq 30 ]; then
17
    echo "API did not become ready at http://localhost:8080/health/ready." >&2
18
    exit 1
19
  fi
20
21
  sleep 2
22
done

Leave the shell open while running CLI commands through the forwarded port. The trap stops the port-forward when the shell exits; to stop it earlier, run kill "$NMP_PORT_FORWARD_PID".

Configure the CLI to use the deployed API:

1
nemo config set --base-url http://localhost:8080
2
nemo workspaces list

Upgrade the chart

Update NMP_HELM_CHART_VERSION, review the release notes for required values changes, and run:

1
helm repo update nemo-platform
2
3
helm upgrade "$NMP_HELM_RELEASE" nemo-platform/nemo-platform \
4
  --version "$NMP_HELM_CHART_VERSION" \
5
  --namespace "$NMP_NAMESPACE" \
6
  -f values.yaml \
7
  --wait \
8
  --timeout 20m

For a release installed from GHCR with NMP_HELM_CHART_REF, use the Helm OCI reference:

1
helm upgrade "$NMP_HELM_RELEASE" "$NMP_HELM_CHART_REF" \
2
  --version "$NMP_HELM_CHART_VERSION" \
3
  --namespace "$NMP_NAMESPACE" \
4
  -f values.yaml \
5
  --wait \
6
  --timeout 20m

Uninstall the platform

To uninstall the Helm release:

1
helm uninstall "$NMP_HELM_RELEASE" -n "$NMP_NAMESPACE"

• PVCs are preserved to prevent accidental data loss. Delete them manually if no longer needed.
• CRDs are not removed by Helm design (upstream issue) to avoid destroying custom resources across the cluster.
• Completed jobs, secrets, and the namespace may also remain.

If you need a complete teardown for CI/CD pipelines or reinstalling in the same namespace, delete the namespace after helm uninstall and inventory CRDs before deleting any cluster-wide API definitions:

1
kubectl delete namespace "$NMP_NAMESPACE"
2
kubectl get crd -o custom-columns=NAME:.metadata.name,API_GROUP:.spec.group

Troubleshooting

Volcano admission webhook blocks pod creation

Symptom: Pod creation fails cluster-wide with an error like:

Internal error occurred: failed calling webhook "mutatepod.volcano.sh":
failed to call webhook: Post "https://volcano-admission-service.volcano-system.svc:443/pods/mutate?timeout=10s":
tls: failed to verify certificate: x509: certificate signed by unknown authority

Cause: The Volcano MutatingWebhookConfiguration registers with failurePolicy: Fail before the volcano-admission-init job finishes generating TLS certificates. This affects all namespaces, not just Volcano workloads.

Fix: Wait for the webhook to become functional, then restart the admission deployment to force certificate regeneration:

1
kubectl rollout restart deployment/volcano-admission -n volcano-system
2
kubectl rollout status deployment/volcano-admission -n volcano-system

Verify the webhook is accepting requests before retrying your Helm install:

1
until kubectl run volcano-webhook-test --image=busybox --restart=Never --dry-run=server -o yaml 2>/dev/null; do
2
  echo "Volcano webhook not ready yet, waiting..."
3
  sleep 5
4
done
5
kubectl delete pod volcano-webhook-test --ignore-not-found=true