nvcrectl setup

nvcrectl setup init

Installs CRE via Helm and its dependencies on the target cluster.

$
nvcrectl setup init [flags]

What it installs

Runs two phases in order:

Use --skip-phases=deps to skip Kubeflow Trainer if it is already installed.

Retry behavior and automatic recovery

setup init converges from any partial state, so re-running it after a failure is always safe to try:

• Already deployed: when the kubeflow-trainer release is deployed at the pinned chart version, the deps phase prints “already deployed” and skips the upgrade entirely. Not re-rendering the chart means not re-rolling its webhook certificates.

• Failed or pending release: the install is attempted once. If it fails with the webhook Secret field-ownership conflict signature (Apply failed with ... conflicts on .data fields of Secrets in kubeflow-system) and the release state agrees (failed or pending-*), setup init performs an automatic recovery: uninstall the release, delete its four CRDs (trainjobs, trainingruntimes, clustertrainingruntimes in trainer.kubeflow.org; jobsets in jobset.x-k8s.io), delete the kubeflow-system namespace, and reinstall the pinned chart. Exactly one recovery attempt is made per run.

• Safety gate: automatic recovery is refused when any TrainJob or JobSet instance exists, or when a TrainingRuntime/ClusterTrainingRuntime exists that is not Helm-owned (missing the app.kubernetes.io/managed-by: Helm label) — deleting the CRDs would destroy them. In that case, and for any ambiguous failure, setup init fails fast and prints the manual procedure:

  $
  helm uninstall kubeflow-trainer --namespace kubeflow-system
  $
  kubectl delete crd trainjobs.trainer.kubeflow.org trainingruntimes.trainer.kubeflow.org \
  >
    clustertrainingruntimes.trainer.kubeflow.org jobsets.jobset.x-k8s.io
  $
  kubectl delete namespace kubeflow-system
  $
  nvcrectl setup init   # reinstalls the pinned Kubeflow Trainer

• Confirmation: in interactive mode the recovery plan is printed and re-confirmed before anything is deleted; --auto-approve covers CI.

Flags

Example

$
# Standard install
$
nvcrectl setup init
$
$
# With GHCR authentication
$
nvcrectl setup init --image-pull-secret $GITHUB_TOKEN
$
$
# Skip Kubeflow Trainer (already installed)
$
nvcrectl setup init --skip-phases=deps

nvcrectl setup status

Reports the installation status of CRE and its dependencies by querying the cluster.

$
nvcrectl setup status [flags]

Components checked: creCRDs, creController, kubeflowTrainer, logProfiles, gpuOperator, dcgm (optional).

The Helm releases managed by setup init (cluster-readiness-engine and kubeflow-trainer) are also checked via the helm CLI and reported under helmReleases. A release in a failed or pending state (e.g. failed, pending-upgrade) makes the status not ready. A release helm has no record of, or that cannot be queried (helm not in PATH), is reported but does not affect readiness.

Flags

Example

$
nvcrectl setup status
$
nvcrectl setup status -o json

nvcrectl setup reset

Removes CRE and its dependencies from the target cluster. Kubeflow Trainer is removed by default.

$
nvcrectl setup reset [flags]

What it removes

Runs three phases in order:

Use --skip-phases=deps to keep Kubeflow Trainer.

After all phases complete, setup reset prints a Retained resources block listing any namespaces or secrets that were not deleted (Helm never removes the release namespace, and the nvcrectl-pull-secret created by setup init is outside the Helm release). Each entry includes the kubectl delete command to remove it manually if you want a pristine cluster.

Flags

Example

$
# Full uninstall (including Kubeflow Trainer)
$
nvcrectl setup reset
$
$
# Keep Kubeflow Trainer
$
nvcrectl setup reset --skip-phases=deps