CLI Reference

Complete reference for the aicr command-line interface.

Overview

AICR provides a four-step workflow for optimizing GPU infrastructure:

┌──────────────┐      ┌──────────────┐      ┌──────────────┐      ┌──────────────┐
│   Snapshot   │─────▶│    Recipe    │─────▶│   Validate   │─────▶│    Bundle    │
└──────────────┘      └──────────────┘      └──────────────┘      └──────────────┘

Step 1: Capture system configuration Step 2: Generate optimization recipes Step 3: Validate constraints against cluster Step 4: Create deployment bundles

Global Flags

Available for all commands:

Logging Modes

AICR supports three logging modes:

1. CLI Mode (default): Minimal user-friendly output

   • Just message text without timestamps or metadata
   • Error messages display in red (ANSI color)
   • Example: Snapshot captured successfully

2. Text Mode (--debug): Debug output with full metadata

   • Key=value format with time, level, source location
   • Example: time=2025-01-06T10:30:00.123Z level=INFO module=aicr version=v1.0.0 msg="snapshot started"

3. JSON Mode (--log-json): Structured JSON for automation

   • Machine-readable format for log aggregation
   • Example: \{"time":"2025-01-06T10:30:00.123Z","level":"INFO","msg":"snapshot started"\}

Examples:

$
# Default: Clean CLI output
$
aicr snapshot
$
$
# Debug mode: Full metadata
$
aicr --debug snapshot
$
$
# JSON mode: Structured logs
$
aicr --log-json snapshot
$
$
# Combine with other flags
$
aicr --debug --output system.yaml snapshot

Commands

aicr snapshot

Capture comprehensive system configuration including OS, GPU, Kubernetes, and SystemD settings.

Synopsis:

$
aicr snapshot [flags]

Flags:

Output Destinations:

• stdout: Default when no -o flag specified
• File: Local file path (/path/to/snapshot.yaml)
• ConfigMap: Kubernetes ConfigMap URI (cm://namespace/configmap-name)

What it captures:

• SystemD Services: containerd, docker, kubelet configurations
• OS Configuration: grub, kmod, sysctl, release info
• Kubernetes: server version, images, ClusterPolicy
• GPU: driver version, CUDA, MIG settings, hardware info
• NodeTopology: node topology (cluster-wide taints and labels across all nodes)

Examples:

$
# Output to stdout (YAML)
$
aicr snapshot
$
$
# Save to file (JSON)
$
aicr snapshot --output system.json --format json
$
$
# Save to Kubernetes ConfigMap (requires cluster access)
$
aicr snapshot --output cm://gpu-operator/aicr-snapshot
$
$
# Debug mode
$
aicr --debug snapshot
$
$
# Table format (human-readable)
$
aicr snapshot --format table
$
$
# With custom kubeconfig
$
aicr snapshot --kubeconfig ~/.kube/prod-cluster
$
$
# Targeting specific nodes
$
aicr snapshot \
>
  --namespace gpu-operator \
>
  --node-selector accelerator=nvidia-h100 \
>
  --node-selector zone=us-west1-a
$
$
# With tolerations for tainted nodes
$
# (By default all taints are tolerated - only needed to restrict tolerations)
$
aicr snapshot \
>
  --toleration nvidia.com/gpu=present:NoSchedule
$
$
# Full example with all options
$
aicr snapshot \
>
  --kubeconfig ~/.kube/config \
>
  --namespace gpu-operator \
>
  --image ghcr.io/nvidia/aicr:v0.8.0 \
>
  --job-name snapshot-gpu-nodes \
>
  --service-account-name aicr \
>
  --node-selector accelerator=nvidia-h100 \
>
  --toleration nvidia.com/gpu:NoSchedule \
>
  --timeout 10m \
>
  --output cm://gpu-operator/aicr-snapshot \
>
  --no-cleanup
$
$
# Custom template formatting
$
aicr snapshot --template examples/templates/snapshot-template.md.tmpl
$
$
# Template with file output
$
aicr snapshot --template examples/templates/snapshot-template.md.tmpl --output report.md
$
$
# With custom template
$
aicr snapshot \
>
  --namespace gpu-operator \
>
  --template examples/templates/snapshot-template.md.tmpl \
>
  --output cluster-report.yaml

Snapshot Config File Mode

Drive aicr snapshot from an AICRConfig document so the snapshot inputs version-control alongside the recipe, bundle, and validate steps in an end-to-end workflow.

1
kind: AICRConfig
2
apiVersion: aicr.nvidia.com/v1alpha1
3
metadata:
4
  name: gke-h100-training
5
spec:
6
  snapshot:
7
    output:
8
      path: snapshot.yaml          # written to disk; same shape as -o
9
      format: yaml                 # yaml | json | table
10
      template: ""                 # optional Go template path
11
    agent:
12
      namespace: aicr-validation
13
      image: ""                    # default: ghcr.io/nvidia/aicr:latest
14
      imagePullSecrets: []
15
      jobName: aicr
16
      serviceAccountName: aicr
17
      nodeSelector:
18
        nodeGroup: gpu-worker
19
      tolerations:
20
        - dedicated=gpu-workload:NoSchedule
21
        - nvidia.com/gpu=present:NoSchedule
22
      requireGpu: false
23
      runtimeClassName: ""         # mutually exclusive with requireGpu
24
      os: ""                       # ubuntu | rhel | cos | amazonlinux | talos
25
      requests: ""                 # "cpu=500m,memory=1Gi"
26
      limits: ""                   # "cpu=1,memory=2Gi"
27
    execution:
28
      timeout: 5m
29
      noCleanup: false
30
      privileged: true             # set false for PSS-restricted namespaces
31
      maxNodesPerEntry: 0          # 0 = unlimited topology entries

Precedence: a CLI flag always wins over the matching config field. Selectors and tolerations omitted entirely inherit the snapshotter’s compiled-in defaults (tolerations defaults to tolerate all taints); an explicit empty list (tolerations: []) clears the tolerate-all default — the same nil-vs-empty semantics used by spec.validate.agent.

$
# Run snapshot driven entirely by config
$
aicr snapshot --config aicr-config.yaml
$
$
# Reuse the same config but write to a one-off path
$
aicr snapshot --config aicr-config.yaml -o /tmp/snapshot.yaml

Custom Templates

The --template flag enables custom output formatting using Go templates with Sprig functions. Templates receive the full Snapshot struct:

1
# Available template data structure:
2
.Kind           # Resource kind ("Snapshot")
3
.APIVersion     # API version string
4
.Metadata       # Map of key-value pairs (timestamp, version, source-node)
5
.Measurements   # Array of Measurement objects
6
  .Type         # Measurement type (K8s, GPU, OS, SystemD, NodeTopology)
7
  .Subtypes     # Array of Subtype objects
8
    .Name       # Subtype name (e.g., "server", "smi", "grub")
9
    .Data       # Map of readings (key -> Reading with .String method)
10
11
# NodeTopology measurement type has subtypes: summary, taint, label
12
# Taint encoding: effect|value|node1,node2,...  (parseable with Sprig splitList "|")
13
# Label encoding: value|node1,node2,...

Example template extracting key cluster info:

1
cluster:
2
  kubernetes: \{\{ with index .Measurements 0 \}\}\{\{ range .Subtypes \}\}\{\{ if eq .Name "server" \}\}
3
    version: \{\{ (index .Data "version").String \}\}\{\{ end \}\}\{\{ end \}\}\{\{ end \}\}
4
  gpu: \{\{ range .Measurements \}\}\{\{ if eq .Type.String "GPU" \}\}\{\{ range .Subtypes \}\}\{\{ if eq .Name "smi" \}\}
5
    model: \{\{ (index .Data "gpu.model").String \}\}
6
    count: \{\{ (index .Data "gpu-count").String \}\}\{\{ end \}\}\{\{ end \}\}\{\{ end \}\}\{\{ end \}\}

See examples/templates/snapshot-template.md.tmpl for a complete example template that generates a concise cluster report.

Agent Deployment Mode

When running against a cluster, AICR deploys a Kubernetes Job to capture the snapshot:

1. Deploys RBAC: ServiceAccount, Role, RoleBinding, ClusterRole, ClusterRoleBinding
2. Creates Job: Runs aicr snapshot as a container on the target node
3. Waits for completion: Monitors Job status with configurable timeout
4. Retrieves snapshot: Reads snapshot from ConfigMap after Job completes
5. Writes output: Saves snapshot to specified output destination
6. Cleanup: Deletes Job and RBAC resources (use --no-cleanup to keep for debugging)

Benefits of agent deployment:

• Capture configuration from actual cluster nodes (not local machine)
• No need to run kubectl manually
• Programmatic deployment for automation/CI/CD
• Reusable RBAC resources across multiple runs

Agent deployment requirements:

• Kubernetes cluster access (via kubeconfig)
• Cluster admin permissions (for RBAC creation)
• GPU nodes with nvidia-smi (for GPU metrics)

ConfigMap Output

When using ConfigMap URIs (cm://namespace/name), the snapshot is stored directly in Kubernetes:

1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
  name: aicr-snapshot
5
  namespace: gpu-operator
6
  labels:
7
    app.kubernetes.io/name: aicr
8
    app.kubernetes.io/component: snapshot
9
    app.kubernetes.io/version: <aicr-version>
10
data:
11
  snapshot.yaml: |
12
    # Full snapshot content
13
  format: yaml
14
  timestamp: "2025-12-31T10:30:00Z"

Snapshot Structure:

1
apiVersion: aicr.nvidia.com/v1alpha1
2
kind: Snapshot
3
metadata:
4
  created: "2025-12-31T10:30:00Z"
5
  hostname: gpu-node-1
6
measurements:
7
  - type: SystemD
8
    subtypes: [...]
9
  - type: OS
10
    subtypes: [...]
11
  - type: K8s
12
    subtypes: [...]
13
  - type: GPU
14
    subtypes: [...]

aicr recipe

Generate optimized configuration recipes from query parameters or captured snapshots.

Synopsis:

$
aicr recipe [flags]

Modes:

Config File Mode (Recommended)

Generate recipes using an AICRConfig document. The same file format also drives the bundle command, so a single file can describe an end-to-end recipe-to-bundle workflow.

Flags:

The config file uses a Kubernetes-style envelope:

1
kind: AICRConfig
2
apiVersion: aicr.nvidia.com/v1alpha1
3
metadata:
4
  name: gb200-eks-ubuntu-training
5
spec:
6
  recipe:
7
    criteria:
8
      service: eks
9
      os: ubuntu
10
      accelerator: gb200
11
      intent: training
12
      nodes: 8
13
    output:
14
      path: recipe.yaml
15
      format: yaml

Individual CLI flags always override config file values. For slice/map flags, presence on the CLI replaces the file’s value (no append).

$
# Load criteria from config file
$
aicr recipe --config config.yaml
$
$
# Override service from file
$
aicr recipe --config config.yaml --service gke
$
$
# Save output to file
$
aicr recipe --config config.yaml -o recipe.yaml
$
$
# Load config from a URL (e.g. CI shared template)
$
aicr recipe --config https://team.example.com/configs/eks-h100-training.yaml

--config accepts a local file path or an HTTP/HTTPS URL. ConfigMap (cm://) sources are not supported; export the data with kubectl get cm <name> -o yaml and pass the resulting file.

Query Mode

Generate recipes using direct system parameters:

Flags:

Service / Accelerator / OS / Intent / Platform value listings above are the OSS-embedded set. When --data registers additional values (e.g., undisclosed providers, proprietary platforms), the CLI admits them at runtime through the criteria registry — see Data Extension. --criteria-strict restores the OSS-only set regardless of what --data contributes.

Examples:

$
# Basic recipe for Ubuntu on EKS with H100
$
aicr recipe --os ubuntu --service eks --accelerator h100
$
$
# Training workload with multiple GPU nodes
$
aicr recipe \
>
  --service eks \
>
  --accelerator gb200 \
>
  --intent training \
>
  --os ubuntu \
>
  --nodes 8 \
>
  --format yaml
$
$
# Kubeflow training workload
$
aicr recipe \
>
  --service eks \
>
  --accelerator h100 \
>
  --intent training \
>
  --os ubuntu \
>
  --platform kubeflow
$
$
# Save to file (--gpu is an alias for --accelerator)
$
aicr recipe --os ubuntu --gpu h100 --output recipe.yaml

Snapshot Mode

Generate recipes from captured snapshots:

Flags:

Snapshot Sources:

• File: Local file path (./snapshot.yaml)
• URL: HTTP/HTTPS URL (https://example.com/snapshot.yaml)
• ConfigMap: Kubernetes ConfigMap URI (cm://namespace/configmap-name)

Examples:

$
# Generate recipe from local snapshot file
$
aicr recipe --snapshot system.yaml --intent training
$
$
# From ConfigMap (requires cluster access)
$
aicr recipe --snapshot cm://gpu-operator/aicr-snapshot --intent training
$
$
# From ConfigMap with custom kubeconfig
$
aicr recipe \
>
  --snapshot cm://gpu-operator/aicr-snapshot \
>
  --kubeconfig ~/.kube/prod-cluster \
>
  --intent training
$
$
# Output to ConfigMap
$
aicr recipe -s system.yaml -o cm://gpu-operator/aicr-recipe
$
$
# Chain snapshot → recipe with ConfigMaps
$
aicr snapshot -o cm://default/snapshot
$
aicr recipe -s cm://default/snapshot -o cm://default/recipe
$
$
# With custom output
$
aicr recipe -s system.yaml -i inference -o recipe.yaml --format yaml

Output structure:

1
apiVersion: aicr.nvidia.com/v1alpha1
2
kind: Recipe
3
metadata:
4
  version: v1.0.0
5
  created: "2025-12-31T10:30:00Z"
6
  appliedOverlays:
7
    - base
8
    - eks
9
    - eks-training
10
    - gb200-eks-training
11
criteria:
12
  service: eks
13
  accelerator: gb200
14
  intent: training
15
  os: any
16
componentRefs:
17
  - name: gpu-operator
18
    version: v25.3.3
19
    order: 1
20
    repository: https://helm.ngc.nvidia.com/nvidia
21
constraints:
22
  driver:
23
    version: "580.82.07"
24
    cudaVersion: "13.1"

aicr query

Query a specific value from the fully hydrated recipe configuration. Resolves a recipe from criteria (same as aicr recipe), merges all base, overlay, and inline value overrides, then extracts the value at the given dot-path selector.

Synopsis:

$
aicr query --selector <path> [flags]

Flags:

All aicr recipe flags are supported, plus:

Selector Syntax

Uses dot-delimited paths consistent with Helm --set and yq:

Leading dots are optional (yq-style): .components.gpu-operator.chart and components.gpu-operator.chart are equivalent.

Output:

• Scalar values (string, number, bool) are printed as plain text — no YAML wrapper
• Complex values (maps, lists) are printed as YAML (default) or JSON (--format json)

Examples:

$
# Get a specific Helm value
$
aicr query --service eks --accelerator h100 --intent training \
>
  --selector components.gpu-operator.values.driver.version
$
# stdout: 570.86.16
$
$
# Get a value subtree
$
aicr query --service eks --accelerator h100 --intent training \
>
  --selector components.gpu-operator.values.driver
$
# stdout:
$
#   version: "570.86.16"
$
#   repository: nvcr.io/nvidia
$
$
# Get the full hydrated component
$
aicr query --service eks --accelerator h100 --intent training \
>
  --selector components.gpu-operator
$
$
# Get deployment order
$
aicr query --service eks --accelerator h100 --intent training \
>
  --selector deploymentOrder
$
$
# Use in shell scripts
$
VERSION=$(aicr query --service eks --accelerator h100 --intent training \
>
  --selector components.gpu-operator.values.driver.version)
$
echo "Driver version: $VERSION"
$
$
# JSON output for complex values
$
aicr query --service eks --accelerator h100 --intent training \
>
  --selector components.gpu-operator.values --format json
$
$
# Query from snapshot
$
aicr query --snapshot snapshot.yaml \
>
  --selector components.gpu-operator.values.driver.version
$
$
# Full hydrated recipe
$
aicr query --service eks --accelerator h100 --intent training --selector .

Advanced Examples:

$
# Cross-cloud comparison: how Prometheus storage differs between EKS and GKE
$
# EKS provisions a 50Gi persistent EBS volume (gp2)
$
aicr query --service eks --intent training \
>
  --selector components.kube-prometheus-stack.values.prometheus.prometheusSpec.storageSpec
$
# GKE uses a 10Gi ephemeral emptyDir (GMP handles long-term retention)
$
aicr query --service gke --intent training \
>
  --selector components.kube-prometheus-stack.values.prometheus.prometheusSpec.storageSpec
$
$
# Compare deployment order across clouds
$
# EKS deploys 12 components (includes aws-ebs-csi-driver, aws-efa, nodewright-customizations)
$
aicr query --service eks --accelerator h100 --intent training --selector deploymentOrder
$
# GKE deploys 9 components (storage and networking are platform-managed)
$
aicr query --service gke --accelerator h100 --intent training --selector deploymentOrder
$
$
# Pin the exact driver version into Terraform/Pulumi variables
$
DRIVER_VERSION=$(aicr query --service eks --accelerator h100 --intent training \
>
  --selector components.gpu-operator.values.driver.version)
$
echo "gpu_driver_version = \"$\{DRIVER_VERSION\}\""
$
$
# Compare nodewright tuning parameters across accelerators
$
# H100: real tuning packages (kernel setup, nvidia-tuned, full setup)
$
aicr query --service eks --accelerator h100 --intent training \
>
  --selector components.nodewright-customizations.values
$
# GB200: same value structure, but manifest renders a no-op (ARM64 packages pending)
$
aicr query --service eks --accelerator gb200 --intent training \
>
  --selector components.nodewright-customizations.values
$
$
# Watch constraints tighten as you add specificity
$
# Just "EKS" → 1 constraint (K8s >= 1.28)
$
aicr query --service eks --selector constraints
$
# Add GPU + intent + OS → 4 constraints (K8s >= 1.32.4, Ubuntu 24.04, kernel >= 6.8)
$
aicr query --service eks --accelerator h100 --intent training --os ubuntu \
>
  --selector constraints

aicr validate

Validate a system snapshot against the constraints defined in a recipe to verify cluster compatibility. Supports multi-phase validation with different validation stages.

For a task-oriented walkthrough (capture snapshot → generate recipe → run each phase, with worked training and inference examples), see Validation.

Synopsis:

$
aicr validate [flags]

Flags:

Input Sources:

• File: Local file path (./recipe.yaml, ./snapshot.yaml)
• URL: HTTP/HTTPS URL (https://example.com/recipe.yaml)
• ConfigMap: Kubernetes ConfigMap URI (cm://namespace/configmap-name)

Validation Phases

Validation can be run in different phases to validate different aspects of the deployment:

Note: Readiness constraints (K8s version, OS, kernel) are always evaluated implicitly before any phase runs. If readiness fails, validation stops before deploying any Jobs.

Deployment phase checks:

The deployment phase verifies that the cluster is actually ready for GPU workloads — not just that install commands returned successfully. It covers:

• Enabled component namespaces are Active.
• Declared expectedResources (Deployments, DaemonSets, etc.) exist and are healthy.
• When nodewright-customizations is enabled: every Skyhook CR the recipe declares reports status.status == complete. The set of expected CR names is extracted from the recipe’s own ComponentRef.ManifestFiles for this component, so unrelated Skyhook CRs on the cluster (stale from prior deploys, or owned by another tenant) are ignored. If no Skyhook names can be extracted from those ManifestFiles, deployment validation fails closed as a recipe/configuration error instead of skipping.
• When gpu-operator is enabled: ClusterPolicy reports status.state == ready.
• When nvidia-dra-driver-gpu is enabled: the kubelet-plugin DaemonSet is ready. Discovery is by the upstream chart’s role-suffix convention — the validator finds the single DaemonSet in the component namespace whose name ends in -kubelet-plugin, so custom fullnameOverride values are handled automatically.

Graceful skip: If a component is declared in the recipe but its CRD is not yet registered on the cluster (e.g., fresh cluster, operator chart not installed), the corresponding readiness check is skipped rather than failing. Once the CRD is present, the check runs and a missing CR is treated as a real failure — for example, if the gpu-operator CRD is registered but no ClusterPolicy CR exists, deployment validation fails with a “CR missing” diagnostic rather than silently passing. Other errors still fail closed: an RBAC denial on skyhooks.skyhook.nvidia.com returns HTTP 403 (not a NoMatch), so the validator surfaces it as a failure instead of silently skipping the Skyhook check.

Day-N re-verification: Because this is a read-only check against live cluster state, re-running aicr validate --phase deployment after scale-up, upgrade, or other runtime changes is safe and answers the same “is this cluster ready for GPU workloads now?” question.

Phase Dependencies:

• Phases run sequentially when using --phase all
• If a phase fails, subsequent phases are skipped
• Use individual phases for targeted validation during specific deployment stages

Constraint Format

Constraints use fully qualified measurement paths: \{Type\}.\{Subtype\}.\{Key\}

Supported Operators

Examples:

$
# Validate snapshot against recipe (readiness constraints run implicitly)
$
aicr validate --recipe recipe.yaml --snapshot snapshot.yaml
$
$
# Validate specific phase
$
aicr validate \
>
  --recipe recipe.yaml \
>
  --snapshot snapshot.yaml \
>
  --phase deployment
$
$
# Run all validation phases
$
aicr validate \
>
  --recipe recipe.yaml \
>
  --snapshot snapshot.yaml \
>
  --phase all
$
$
# Load snapshot from ConfigMap
$
aicr validate \
>
  --recipe recipe.yaml \
>
  --snapshot cm://gpu-operator/aicr-snapshot
$
$
# Save results to file
$
aicr validate \
>
  --recipe recipe.yaml \
>
  --snapshot cm://gpu-operator/aicr-snapshot \
>
  --output validation-results.json
$
$
# Validate deployment phase after components are installed
$
aicr validate \
>
  --recipe recipe.yaml \
>
  --snapshot snapshot.yaml \
>
  --phase deployment
$
$
# Run performance validation
$
aicr validate \
>
  --recipe recipe.yaml \
>
  --snapshot snapshot.yaml \
>
  --phase performance
$
$
# With custom kubeconfig
$
aicr validate \
>
  --recipe recipe.yaml \
>
  --snapshot cm://gpu-operator/aicr-snapshot \
>
  --kubeconfig ~/.kube/prod-cluster
$
$
# Write a recipe-evidence v1 attestation bundle (unsigned, on disk).
$
# --bom is optional: when omitted, aicr synthesizes a recipe-bound BOM from
$
# the recipe's component refs and validator catalog images.
>
aicr validate \
>
  --recipe recipe.yaml --snapshot snapshot.yaml \
>
  --emit-attestation ./out
>
# Writes ./out/summary-bundle/ and ./out/pointer.yaml.
>
>
# Use an exhaustive BOM (e.g., `make bom`-produced) instead of the auto-generated one
>
aicr validate \
>
  --recipe recipe.yaml --snapshot snapshot.yaml \
>
  --emit-attestation ./out --bom dist/bom/bom.cdx.json
>
>
# Sign and push a recipe-evidence bundle to OCI (cosign keyless via Sigstore public-good).
>
# Token acquisition follows the same precedence chain as `aicr bundle --attest`:
>
# pre-fetched COSIGN_IDENTITY_TOKEN > ambient GitHub Actions OIDC > --oidc-device-flow > interactive browser.
>
aicr validate \
>
  --recipe recipe.yaml --snapshot snapshot.yaml \
>
  --emit-attestation ./out \
>
  --push ghcr.io/myorg/aicr-evidence
>
# After this, copy ./out/pointer.yaml to recipes/evidence/<recipe>.yaml
>
>
# Validate on a cluster with custom GPU node labels (non-standard labels that AICR doesn't
$
# recognize by default, e.g., using a custom node pool label instead of cloud-provider defaults)
$
aicr validate \
>
  --recipe recipe.yaml \
>
  --node-selector my-org/gpu-pool=true \
>
  --phase performance
$
$
# Override both node selector and tolerations for a non-standard taint setup
$
aicr validate \
>
  --recipe recipe.yaml \
>
  --node-selector gpu-type=h100 \
>
  --toleration gpu-type=h100:NoSchedule

Validate Config File Mode

aicr validate --config <path> reads inputs from an AICRConfig YAML/JSON file under spec.validate. CLI flags always override values loaded from --config; override events are logged at INFO so users can see which input won. The OIDC identity token used for --push signing stays out of the schema by design (short-lived tokens must not be committed); the CLI resolves it at sign time through the precedence chain described on --identity-token.

Supported schema:

1
kind: AICRConfig
2
apiVersion: aicr.nvidia.com/v1alpha1
3
metadata:
4
  name: prod-validate
5
spec:
6
  validate:
7
    input:
8
      recipe: ./recipe.yaml
9
      snapshot: ./snapshot.yaml          # optional; omit to capture live
10
    agent:                               # only used when input.snapshot is empty
11
      namespace: aicr-validation
12
      image: ghcr.io/nvidia/aicr:v0.1.0
13
      imagePullSecrets: [registry-secret]
14
      jobName: aicr-validate
15
      serviceAccountName: aicr
16
      nodeSelector:
17
        my-org/gpu-pool: "true"
18
      tolerations:
19
        - "gpu-type=h100:NoSchedule"
20
      requireGpu: true
21
    execution:
22
      phases: [deployment, conformance]
23
      failOnError: true                  # default true; set false to report only
24
      noCluster: false
25
      noCleanup: false
26
      timeout: 10m
27
    evidence:
28
      cncf:                              # --evidence-dir / --cncf-submission / --feature
29
        dir: ./out/cncf
30
        cncfSubmission: false
31
        features: []                     # empty = all features
32
      attestation:                       # --emit-attestation / --bom / --push / ...
33
        out: ./out/attestation
34
        bom: dist/bom/bom.cdx.json       # optional; auto-generated from recipe + validators when absent
35
        push: ghcr.io/myorg/aicr-evidence
36
        plainHTTP: false
37
        insecureTLS: false

Examples:

$
# Use a config file
$
aicr validate --config validate.yaml
$
$
# Override a single config value from the CLI
$
aicr validate --config validate.yaml --phase deployment
$
$
# Validate the same recipe across two clusters using two different agent
$
# configs (config-bound) without retyping flags
$
aicr validate --config validate-cluster-a.yaml
$
aicr validate --config validate-cluster-b.yaml

Workload Scheduling

The --node-selector and --toleration flags control scheduling for validation workloads — the inner pods that validators create to test cluster functionality (e.g., NCCL benchmark workers, conformance test pods). They do not affect the validator orchestrator Job, which runs lightweight check logic and is placed on CPU-preferred nodes automatically.

When --node-selector is provided, it replaces the platform-specific selectors that validators use by default:

When --toleration is provided, it replaces the default tolerate-all policy (operator: Exists) on workloads that need to land on tainted GPU nodes.

Validators that use nodeName pinning (nvidia-smi, DRA isolation test) or DRA ResourceClaims for placement (gang scheduling) are not affected by these flags.

Output Structure (CTRF JSON):

Results are output in CTRF (Common Test Report Format) — an industry-standard schema for test reporting.

1
\{
2
  "reportFormat": "CTRF",
3
  "specVersion": "0.0.1",
4
  "timestamp": "2026-03-10T20:10:44Z",
5
  "generatedBy": "aicr",
6
  "results": \{
7
    "tool": \{
8
      "name": "aicr",
9
      "version": "v0.10.3-next"
10
    \},
11
    "summary": \{
12
      "tests": 16,
13
      "passed": 13,
14
      "failed": 0,
15
      "skipped": 3,
16
      "pending": 0,
17
      "other": 0,
18
      "start": 1773173400872,
19
      "stop": 1773173799002
20
    \},
21
    "tests": [
22
      \{
23
        "name": "operator-health",
24
        "status": "passed",
25
        "duration": 0,
26
        "suite": ["deployment"],
27
        "stdout": ["Found 1 gpu-operator pod(s)", "Running: 1/1"]
28
      \},
29
      \{
30
        "name": "expected-resources",
31
        "status": "passed",
32
        "duration": 0,
33
        "suite": ["deployment"],
34
        "stdout": ["All deployment resources and required readiness signals are healthy"]
35
      \},
36
      \{
37
        "name": "nccl-all-reduce-bw",
38
        "status": "passed",
39
        "duration": 234000,
40
        "suite": ["performance"],
41
        "stdout": ["NCCL All Reduce bandwidth: 488.37 GB/s", "Constraint: >= 100 → true"]
42
      \},
43
      \{
44
        "name": "inference-perf",
45
        "status": "passed",
46
        "duration": 612000,
47
        "suite": ["performance"],
48
        "stdout": [
49
          "RESULT: Inference throughput: 37961.24 tokens/sec",
50
          "RESULT: Inference TTFT p99: 146.30 ms",
51
          "Throughput constraint: >= 5000 → PASS",
52
          "TTFT p99 constraint: <= 200 → PASS"
53
        ]
54
      \},
55
      \{
56
        "name": "dra-support",
57
        "status": "passed",
58
        "duration": 8000,
59
        "suite": ["conformance"],
60
        "stdout": ["DRA GPU allocation successful"]
61
      \},
62
      \{
63
        "name": "cluster-autoscaling",
64
        "status": "skipped",
65
        "duration": 0,
66
        "suite": ["conformance"],
67
        "stdout": ["SKIP reason=\"Karpenter not found\""]
68
      \}
69
    ]
70
  \}
71
\}

Note: The tests array above is truncated for brevity. A full validation run produces one entry per check across all phases. Each entry includes stdout with detailed diagnostic output.

Test Statuses:

Exit Codes:

aicr diff

Compare two snapshots field-by-field to surface configuration drift between cluster states. Reports added, removed, and modified readings across every measurement type (K8s, GPU, OS, SystemD, NodeTopology).

Synopsis:

$
aicr diff --baseline <path|cm://...> --target <path|cm://...> [flags]

Flags:

Inputs:

• File paths (./baseline.yaml, /tmp/snap.json)
• ConfigMap URIs (cm://gpu-operator/aicr-snapshot)
• Both inputs may mix freely; e.g., a local baseline file vs. a live ConfigMap target.

Output Semantics:

• A nil reading is rendered as the literal <nil> so it cannot be confused with an empty-string value (""). Both forms surface as drift when one side is nil and the other is a concrete value.
• Changes are emitted in deterministic order (sorted by Path) so the diff is reproducible across runs and machines.
• The Result envelope includes baselineSource and targetSource (the supplied paths), a changes array, and a summary with added, removed, modified, and total counts.

Examples:

$
# Local-file diff in default YAML
$
aicr diff --baseline before.yaml --target after.yaml
$
$
# Human-readable table to stdout
$
aicr diff -b before.yaml --target after.yaml --format table
$
$
# CI/CD gate: non-zero exit on drift, JSON to a file
$
aicr diff -b before.yaml --target after.yaml \
>
  --format json --output drift.json --fail-on-drift
$
$
# Compare two ConfigMaps in the cluster
$
aicr diff \
>
  --baseline cm://gpu-operator/aicr-snapshot-baseline \
>
  --target   cm://gpu-operator/aicr-snapshot
$
$
# Mix file + ConfigMap (golden baseline vs live cluster)
$
aicr diff --baseline ./golden.yaml --target cm://default/aicr-snapshot

Exit Codes:

Note on CI gating: A non-zero exit identifies that drift was detected, but doesn’t by itself distinguish drift from malformed input — both map to exit 2. To differentiate without relying on stderr format (text by default; JSON only with --log-json), inspect the diff payload directly: write the result with --output drift.json --format json and branch on the presence of the file plus its summary.total field. That signal is format-stable regardless of logging mode.

aicr bundle

Generate deployment-ready bundles from recipes containing Helm values, manifests, scripts, and documentation.

Synopsis:

$
aicr bundle [flags]

Flags:

Bundle Config File Mode

The bundle command accepts the same AICRConfig format used by aicr recipe. A single file can populate both spec.recipe and spec.bundle, capturing an end-to-end workflow that can be committed to git, fetched from CI, or shared across environments.

When both spec.recipe.output.path and spec.bundle.input.recipe are set, they must reference the same path; otherwise loading fails fast.

1
kind: AICRConfig
2
apiVersion: aicr.nvidia.com/v1alpha1
3
spec:
4
  bundle:
5
    input:
6
      recipe: ./recipe.yaml
7
    output:
8
      target: oci://ghcr.io/example/bundle:v1.0.0
9
    deployment:
10
      deployer: argocd
11
      repo: https://example.git/charts
12
      set:
13
        - gpuoperator:driver.version=570.86.16
14
    scheduling:
15
      systemNodeSelector:
16
        role: system
17
      acceleratedNodeTolerations:
18
        - "nvidia.com/gpu=present:NoSchedule"
19
      nodes: 8
20
      storageClass: gp3
21
    attestation:
22
      enabled: false
23
    registry:
24
      insecureTLS: false
25
      plainHTTP: false

$
# Drive the bundle entirely from a config file
$
aicr bundle --config bundle.yaml
$
$
# Override the deployer for a one-off run
$
aicr bundle --config bundle.yaml --deployer helm

CLI flags always override values loaded from --config. For slice/map flags (--set, --dynamic, --system-node-selector, etc.), CLI presence replaces the config’s value rather than appending. Override events are logged at INFO so users can see which input won.

Secrets: the cosign identity token is never read from a config file; supply it via --identity-token or COSIGN_IDENTITY_TOKEN.

Node Scheduling

The --accelerated-node-selector and --accelerated-node-toleration flags control scheduling for GPU-specific components:

NFD (Node Feature Discovery) workers must run on all nodes (GPU, CPU, and system) to detect hardware features. This matches the gpu-operator default behavior where NFD workers also run on control-plane nodes. The --accelerated-node-selector is intentionally not applied to NFD workers so they are not restricted to GPU nodes.

Note: When no --accelerated-node-toleration is specified, a default toleration (operator: Exists) is applied to both GPU daemonsets and NFD workers, allowing them to run on nodes with any taint.

Example:

$
aicr bundle --recipe recipe.yaml \
>
  --accelerated-node-selector nodeGroup=gpu-worker \
>
  --accelerated-node-toleration dedicated=worker-workload:NoSchedule \
>
  --accelerated-node-toleration dedicated=worker-workload:NoExecute \
>
  --system-node-selector nodeGroup=system-worker \
>
  --system-node-toleration dedicated=system-workload:NoSchedule \
>
  --system-node-toleration dedicated=system-workload:NoExecute \
>
  --output bundle

Cluster node requirements: This example assumes the cluster has nodes labeled nodeGroup=system-worker with taints dedicated=system-workload:NoSchedule,NoExecute for system infrastructure, and GPU nodes labeled nodeGroup=gpu-worker with taints dedicated=worker-workload:NoSchedule,NoExecute.

This results in:

• GPU daemonsets (driver, device-plugin, toolkit, dcgm): nodeSelector=nodeGroup=gpu-worker + tolerations for dedicated=worker-workload with both NoSchedule and NoExecute
• NFD workers: no nodeSelector (runs on all nodes) + tolerations for dedicated=worker-workload with both NoSchedule and NoExecute
• System components (gpu-operator controller, NFD gc/master, dynamo grove, agentgateway proxy): nodeSelector=nodeGroup=system-worker + tolerations for dedicated=system-workload with both NoSchedule and NoExecute

Behavior:

• All components from the recipe are bundled automatically
• Each component creates a subdirectory in the output directory
• Components are deployed in the order specified by deploymentOrder in the recipe

Storage Class

The --storage-class flag injects a Kubernetes StorageClass name into components at bundle time. StorageClass is a cluster infrastructure detail — the right value depends on what the target cluster has provisioned, not on the recipe.

When provided, the value is written to all Helm value paths declared in the component registry under storageClassPaths, overriding any storageClassName set in recipe overlays. If a per-component --set <component>:<path>=<value> explicitly targets the same path, that value takes precedence over --storage-class.

Example:

$
# Use EBS gp3 instead of the overlay default gp2 on EKS
$
aicr bundle --recipe recipe.yaml \
>
  --storage-class gp3 \
>
  --output bundle
$
$
# Use a custom storage class on an on-prem cluster
$
aicr bundle --recipe recipe.yaml \
>
  --storage-class local-path \
>
  --output bundle

When --storage-class is not set, any storageClassName values already defined in the recipe overlays are preserved as defaults. When it is set, --set <component>:<path>=<value> on the same path still wins — --storage-class only fills in paths that were not explicitly overridden.

If a rendered component creates a PVC at a registry-declared storageClassPaths entry and no usable storageClassName is set after overlay, --storage-class, and --set precedence is resolved, aicr bundle emits a non-blocking warning. The bundle still relies on the target cluster’s default StorageClass in that case.

Deployment Methods

The --deployer flag controls how deployment artifacts are generated:

Note: --dynamic is not supported with --deployer argocd. Use --deployer argocd-helm instead, which produces a Helm chart where all values are overridable at install time.

Deployment Order:

All deployers respect the deploymentOrder field from the recipe, ensuring components are installed in the correct sequence:

• Helm: Components listed in README in deployment order
• Argo CD: Uses argocd.argoproj.io/sync-wave annotation (0 = first, 1 = second, etc.)
• Flux: Uses dependsOn references in HelmRelease CRs (each component depends on the previous component’s terminal release — its <prev>-post release when post-manifests are present, otherwise <prev>). Components with pre-manifests insert a <name>-pre release that the primary HelmRelease depends on, so the chain becomes previous → <name>-pre → <name> → <name>-post → next. The bundle’s root kustomization.yaml is a plain Kustomize file (not a Flux Kustomization CR).
• Helmfile: Uses needs: references in each release (each component depends on its predecessor)

Value Overrides

Override any value in the generated bundle files using dot notation:

$
--set bundler:path.to.field=value

Format: bundler:path=value where:

• bundler - Bundler name (e.g., gpuoperator, networkoperator, certmanager, nodewright-operator, nvsentinel)
• path - Dot-separated path to the field
• value - New value to set

Behavior:

• Duplicate keys: When the same bundler:path is specified multiple times, the last value wins
• Array values: Individual array elements cannot be overridden (no [0] index syntax). Arrays can only be replaced entirely via recipe overrides, not via --set flags. Use recipe-level overrides in componentRefs[].overrides if you need to replace an entire array.
• Type conversion: String values are automatically converted to appropriate types (true/false → bool, numeric strings → numbers)
• Component enable/disable: The special enabled key controls whether a component is included in the bundle. --set <component>:enabled=false excludes the component; --set <component>:enabled=true re-enables a recipe-disabled component. The enabled key is consumed by the bundler and not passed to Helm chart values.

Examples:

$
# Generate all bundles
$
aicr bundle --recipe recipe.yaml --output ./bundles
$
$
# Override values in GPU Operator bundle
$
aicr bundle -r recipe.yaml \
>
  --set gpuoperator:gds.enabled=true \
>
  --set gpuoperator:driver.version=570.86.16 \
>
  -o ./bundles
$
$
# Override multiple components
$
aicr bundle -r recipe.yaml \
>
  --set gpuoperator:mig.strategy=mixed \
>
  --set networkoperator:rdma.enabled=true \
>
  --set networkoperator:sriov.enabled=true \
>
  -o ./bundles
$
$
# Override cert-manager resources
$
aicr bundle -r recipe.yaml \
>
  --set certmanager:controller.resources.memory.limit=512Mi \
>
  --set certmanager:webhook.resources.cpu.limit=200m \
>
  -o ./bundles
$
$
# Override Nodewright manager resources
$
aicr bundle -r recipe.yaml \
>
  --set nodewright-operator:manager.resources.cpu.limit=500m \
>
  --set nodewright-operator:manager.resources.memory.limit=256Mi \
>
  -o ./bundles
$
$
# Disable a component at bundle time (e.g., EBS CSI already installed as EKS addon)
$
aicr bundle -r recipe.yaml \
>
  --set awsebscsidriver:enabled=false \
>
  -o ./bundles
$
$
# Schedule system components on specific node pool
$
aicr bundle -r recipe.yaml \
>
  --system-node-selector nodeGroup=system-pool \
>
  --system-node-toleration dedicated=system:NoSchedule \
>
  -o ./bundles
$
$
# Schedule GPU workloads on labeled GPU nodes
$
aicr bundle -r recipe.yaml \
>
  --accelerated-node-selector nvidia.com/gpu.present=true \
>
  --accelerated-node-toleration nvidia.com/gpu=present:NoSchedule \
>
  -o ./bundles
$
$
# Combined: separate system and GPU scheduling
$
aicr bundle -r recipe.yaml \
>
  --system-node-selector nodeGroup=system-pool \
>
  --system-node-toleration dedicated=system:NoSchedule \
>
  --accelerated-node-selector accelerator=nvidia-h100 \
>
  --accelerated-node-toleration nvidia.com/gpu=present:NoSchedule \
>
  -o ./bundles
$
$
# Set estimated GPU node count (writes to nodeCountPaths in registry)
$
aicr bundle -r recipe.yaml --nodes 8 -o ./bundles
$
$
# Day 2 options: workload-gate and workload-selector for nodewright
$
aicr bundle -r recipe.yaml \
>
  --workload-gate skyhook.nvidia.com/runtime-required=true:NoSchedule \
>
  --workload-selector workload-type=training \
>
  -o ./bundles
$
$
# Generate an attested bundle (opens browser for OIDC auth)
$
aicr bundle -r recipe.yaml --attest -o ./bundles
$
$
# In GitHub Actions (OIDC token detected automatically)
$
aicr bundle -r recipe.yaml --attest -o ./bundles
$
$
# Generate Argo CD Application manifests for GitOps
$
aicr bundle -r recipe.yaml --deployer argocd -o ./bundles
$
$
# Argo CD with Git repository URL (avoids placeholder in app-of-apps.yaml)
$
aicr bundle -r recipe.yaml --deployer argocd \
>
  --repo https://github.com/my-org/my-gitops-repo.git \
>
  -o ./bundles
$
$
# Combine deployer with value overrides
$
aicr bundle -r recipe.yaml \
>
  --deployer argocd \
>
  -o ./bundles