kubectl aicr

The kubectl aicr plugin is how approvers inspect, approve and revoke rollouts, and how anyone can check a plan’s bundle against the AICR CLI. It talks to the API server only; it never calls the operator.

Install

Each AICR GitHub release carries kubectl-aicr-<os>-<arch> for Linux and macOS on amd64 and arm64:

VERSION="<release>"   # the AICR release of your operator chart, without the leading v
gh release download "v${VERSION}" --repo NVIDIA/aicr --pattern kubectl-aicr-linux-amd64
gh attestation verify kubectl-aicr-linux-amd64 --repo NVIDIA/aicr \
  --signer-workflow NVIDIA/aicr/.github/workflows/on-tag.yaml --source-ref "refs/tags/v${VERSION}" \
  && install -m 0755 kubectl-aicr-linux-amd64 /usr/local/bin/kubectl-aicr

kubectl finds any kubectl-aicr on the PATH. Use the plugin from the same release as the operator.

Usage

kubectl aicr COMMAND ROLLOUT [flags]
kubectl aicr reproduce [PLAN] [flags]
kubectl aicr help

Flags may come before or after the name. Every command accepts every flag:

Each command’s API calls share a 30-second deadline. Output is plain text.

describe

kubectl aicr describe rollout-3f9a1c0d2b7e4a51-1

Prints what an approver decides on:

• the rollout, its runtime UID, and the transition (<from> -> <to>, or none (initial install));
• its phase and outcome, deadline, approval (required, pending, approved by … until …, or the revocation), message, and the size and digest of its starting inventory;
• every step: release, operation (Install, Upgrade, Verify, Remove, Adopt), AICR’s assessment, whether it changes CRDs, and its state (in flight, Ready, Failed, …);
• AICR’s upgrade-check report from the operator namespace, verified against the digest the rollout records: each component’s change, versions, verdict and notes. Without one it prints Assessment: none;
• under “Recorded steps”, each transition record’s precondition (before:) and steps. The operator performs only the apply; approving acknowledges the steps before it, and acknowledge the steps after it;
• each acknowledgement already given (Acknowledged: <release> by <user>).

acknowledge

kubectl aicr acknowledge rollout-3f9a1c0d2b7e4a51-1

For a rollout waiting in AwaitingAcknowledgement: records, as you, that the post-apply steps of the release it waits on are done (for NVSentinel, a tripped circuit breaker reset and the CRD verified), and the rollout continues. It refuses a rollout that is not waiting. Approvers only; the username is the one the API server reports.

approve

kubectl aicr approve rollout-3f9a1c0d2b7e4a51-1 --expires 2h --reason "Maintenance window"

Asks the API server who you are (a SelfSubjectReview), then sets the rollout’s spec.approval to your username, the expiry and the reason. Prints rollout <name> approved by <user> until <time>.

Without --expires, the approval lasts until the rollout’s deadline. It refuses a rollout that has ended, is already approved, or needs no approval, and without --expires one whose deadline has passed. An approval can be neither changed nor extended: if it expires, the step in flight finishes, the rollout ends Expired, and a retry needs a new approval.

revoke

kubectl aicr revoke rollout-3f9a1c0d2b7e4a51-1 --reason "Training job not checkpointed"

Records your username, the time and the reason as the revocation. The step in flight finishes, no further step starts, and the rollout ends Revoked. It refuses a rollout that is not approved, already revoked, or has ended; the API server also refuses one in Validating.

reproduce

kubectl aicr reproduce                      # the active plan, else the candidate
kubectl aicr reproduce "sha256:<digest>"    # or a plan ConfigMap name, aicr-plan-<16 hex>

Writes the plan’s recipe to <dir>/recipe.json and prints the plan, the AICR image it was rendered with, the operator’s bundle reference, and three commands to run yourself: a docker run of that AICR image rendering the recipe with the plan’s flags, an oras pull of the operator’s bundle, and a diff -r of the two. Zero diff means the operator deployed exactly what the AICR CLI renders (verify the bundle yourself).

It changes nothing in the cluster. It refuses a plan rebuilt from Applications (it has no recipe), a plan that does not record its render flags, and a recipe whose content does not match its recorded digest.

Permissions

The chart’s aicr-operator-approver ClusterRole and Role grant all of these to the users and groups in its approvers value.