Skip to navigation
Getting Started

Install

View as Markdown

Prerequisites

  • Kubernetes 1.29+ cluster with GPU nodes (the Certification CRD’s CEL rules use the quantity() library, which the API server accepts in new CRD expressions from 1.29)
  • kubectl configured and pointing at the target cluster
  • NVIDIA GPU Operator installed
  • Helm 3.x

Install the CLI

Download and run the installer. It detects your OS and architecture and resolves the newest stable release itself:

curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash

To pin a version, download the installer from that release and pass the tag:

curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/download/<tag>/installer | bash -s -- -v <tag>

The installer verifies the binary against that release’s Sigstore bundle (<binary>.sigstore.json) before installing it, using cosign from your PATH or a pinned copy it downloads and checks. It also checks checksums.txt. If either check fails or cannot run, it stops without installing. --skip-verify skips only the signature check; the checksum check has no override. The installer fetches everything from github.com and runs cosign without --trusted-root, so it does not work on an air-gapped host. There, download the binary and its .sigstore.json on a connected machine, verify them offline as described in Air-gapped verification, and copy the binary into place yourself. See SECURITY.md for what each check proves.

Verify the installation:

nvcrectl --version

Set up the cluster

nvcrectl setup init installs the controller and its dependencies in two phases:

  1. deps — Kubeflow Trainer (required for TrainJob workloads)
  2. helm — NVCRE Helm chart (CRDs, controller deployment, built-in LogProfiles)
nvcrectl setup init

Setup detects the supported JobSet controller ownership pattern. A verified external JobSet controller is preserved and Trainer is installed without its bundled JobSet subchart. Ambiguous ownership stops before Helm mutation and prints an operator-managed fallback. An existing Trainer release with a missing JobSet CRD also stops: restore a compatible CRD using the intended JobSet installation procedure before rerunning setup. Existing JobSet CRDs are retained and are not automatically refreshed. For operator-managed Trainer or JobSet, you are responsible for keeping both CRD schemas, their controllers, and external consumers compatible. Automatic reconciliation of the shared JobSet CRD is deferred work that needs its own design.

Registry access

The controller image and Helm chart are pulled anonymously from GHCR; no token is needed. If your cluster pulls from a private mirror or fork instead, pass --image-pull-secret <github-token> and the CLI creates the pull secret for you.

On clusters with restricted egress, both Helm charts can be pulled from a mirror registry with --chart-ref and --trainer-chart-ref; see Deployment for the full procedure.

Verify

Check that the controller is running:

kubectl get pods -n nvcre

Check that the CRDs are installed:

kubectl get crds | grep nvcre.nvidia.com

Uninstall

nvcrectl setup reset

This removes all NVCRE custom resources, the controller, NVCRE CRDs, Kubeflow Trainer, and the three Trainer CRDs. The shared JobSet CRD is retained because deleting it destroys JobSets in every namespace. A Trainer uninstall error aborts the dependency cleanup before CRD deletion. To keep Kubeflow Trainer, pass --skip-phases=deps:

nvcrectl setup reset --skip-phases=deps