Volcano

The NeMo Platform Helm chart does not install Volcano. Volcano is optional. It is needed for multi-node jobs on the volcano_job backend (Customizer multi-node). Skip it and set rbac.volcanoEnabled: false if you are not running those jobs.

The chart grants the core controller Volcano RBAC when rbac.volcanoEnabled is true (the default).

Job-level knobs (queue, scheduler_name, plugins) are documented in Manage Jobs. Multi-node NCCL injection still requires Kyverno in addition to Volcano.

Install

Pin the installer to Volcano v1.9.0:

1
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.9.0/installer/volcano-development.yaml
2
kubectl wait --for=condition=complete job/volcano-admission-init -n volcano-system --timeout=120s
3
kubectl rollout status deployment/volcano-admission -n volcano-system

Wait for the admission webhook to finish initializing before installing the platform chart. The webhook registers immediately with failurePolicy: Fail, but TLS certificate generation runs asynchronously. If you proceed before the webhook is ready, pod creation can fail with certificate errors.

Troubleshooting

Volcano admission webhook blocks pod creation

Symptom: Pod creation fails cluster-wide with an error like:

Internal error occurred: failed calling webhook "mutatepod.volcano.sh":
failed to call webhook: Post "https://volcano-admission-service.volcano-system.svc:443/pods/mutate?timeout=10s":
tls: failed to verify certificate: x509: certificate signed by unknown authority

Cause: The Volcano MutatingWebhookConfiguration registers with failurePolicy: Fail before the volcano-admission-init job finishes generating TLS certificates. This affects all namespaces, not just Volcano workloads.

Fix: kubectl rollout restart does not rerun volcano-admission-init or replace volcano-admission-secret. Wait for the init Job first. If the webhook still fails, delete the Job and Secret, re-apply the v1.9.0 installer (or recreate the init Job), wait for the Job to complete, then wait for admission rollout:

1
kubectl wait --for=condition=complete job/volcano-admission-init -n volcano-system --timeout=120s
2
3
# If the webhook still fails with unknown authority:
4
kubectl delete job volcano-admission-init -n volcano-system --ignore-not-found
5
kubectl delete secret volcano-admission-secret -n volcano-system --ignore-not-found
6
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.9.0/installer/volcano-development.yaml
7
kubectl wait --for=condition=complete job/volcano-admission-init -n volcano-system --timeout=120s
8
kubectl rollout status deployment/volcano-admission -n volcano-system

The webhook registers with failurePolicy: Fail before TLS is ready, so pod creation can fail cluster-wide until the secret exists. Verify the webhook is accepting requests before retrying your Helm install:

1
until kubectl run volcano-webhook-test --image=busybox --restart=Never --dry-run=server -o yaml 2>/dev/null; do
2
  echo "Volcano webhook not ready yet, waiting..."
3
  sleep 5
4
done
5
kubectl delete pod volcano-webhook-test --ignore-not-found=true