User Guide

CLI Configuration File (AICRConfig)

View as Markdown

AICRConfig is a Kubernetes-style YAML/JSON document that captures the inputs to the five workflow commands — aicr snapshot, aicr recipe, aicr bundle, aicr validate, and aicr verify — so an end-to-end run version-controls as a single file instead of a shell script full of flags. Each command accepts it through the same --config flag:

aicr snapshot --config aicr-config.yaml
aicr recipe   --config aicr-config.yaml
aicr bundle   --config aicr-config.yaml
aicr validate --config aicr-config.yaml
aicr verify   ./my-bundle --config aicr-config.yaml   # bundle dir is positional

The first four are the producer pipeline; spec.verify is the consumer side, so one document can carry both how an artifact is built and the trust floor a downstream consumer enforces against it.

This page documents the complete document schema in one place. The CLI Reference shows per-command usage in its Snapshot, Recipe, Validate, Bundle, and Verify config-file-mode sections. The schema’s source of truth is pkg/config.

Document Envelope

kind: AICRConfig               # required, exactly this value
apiVersion: aicr.run/v1beta1   # required; the only accepted value
metadata:
  name: gke-h100-training      # optional, identifying only
spec:
  snapshot: {}                 # each section optional —
  recipe: {}                   # at least ONE must be present
  bundle: {}
  validate: {}
  verify: {}

Each spec.* section is optional and each command reads only its own section, so a file may carry just one section or any combination. A document with none of the five sections is rejected.

Reading is per-command; validation is not. Every section present in the file is checked when the file loads, whichever command loaded it — so a malformed spec.bundle fails aicr snapshot --config too, naming the spec path that is wrong. Keep that in mind for a single document spanning several sections: an error can name a section the running command never reads.

AICRConfig is an authored file, so its apiVersion is yours to set. The value is aicr.run/v1beta1, and as of v1.0.0 it is the only one accepted: the superseded aicr.run/v1alpha2 was read-with-a-warning in v0.22 and is rejected now, as are empty and unknown values. A config still carrying it must be edited by hand — AICR has no conversion layer. The full release-by-release table is in Catalog and binary compatibility; the policy behind it is ADR-022.

Loading, Precedence, and Secrets

Sources. --config accepts a local file path or an HTTP/HTTPS URL (format detected from the extension; fetches are timeout- and size-bounded). ConfigMap cm:// URIs are intentionally rejected — extract the data with kubectl and pass the resulting file.

Precedence. A CLI flag always wins over the matching config field. For slice and map fields (tolerations, selectors, --set), a flag given on the command line replaces the file’s value; it does not append.

Nil vs. empty. For agent selectors and tolerations, omitting the field entirely (nil) inherits the compiled-in defaults (tolerations defaults to tolerate all taints), while an explicit empty value ({} / []) clears the default. Several booleans are tri-state for the same reason: absent means “inherit the CLI default”, an explicit false is an opt-out (spec.validate.execution.failOnError, failFast, spec.snapshot.execution.privileged, spec.recipe.criteriaStrict, spec.validate.evidence.*).

Secrets are never part of the schema. The cosign OIDC identity token used by attestation and evidence push is deliberately absent — supply it via the COSIGN_IDENTITY_TOKEN environment variable or the --identity-token flag.

Complete Example

kind: AICRConfig
apiVersion: aicr.run/v1beta1
metadata:
  name: eks-h100-training
spec:
  snapshot:
    output:
      path: snapshot.yaml            # same shape as -o
      format: yaml                   # yaml | json | table
      template: ""                   # optional Go template path
    agent:                           # in-cluster snapshot Job pod
      namespace: aicr-validation
      image: ""                      # default: matches the CLI version (:latest on dev/-next builds)
      imagePullSecrets: []
      # jobName is an optional PREFIX, not a name — the run ID is always
      # appended. serviceAccountName is exact-if-exists: an existing
      # ServiceAccount of exactly that name is used verbatim and the run
      # then creates and deletes NO RBAC; otherwise it is a prefix too.
      # Omit both to take the run-scoped defaults.
      nodeSelector:
        nodeGroup: gpu-worker
      tolerations:
        - dedicated=gpu-workload:NoSchedule
      requireGpu: false
      runtimeClassName: ""           # mutually exclusive with requireGpu
      os: ""                         # ubuntu | rhel | cos | amazonlinux | ol | talos
      requests: ""                   # "cpu=500m,memory=1Gi"
      limits: ""
    execution:
      timeout: 5m
      noCleanup: false
      privileged: true               # false for PSS-restricted namespaces
      maxNodesPerEntry: 0            # 0 = unlimited topology entries
  recipe:
    criteria:                        # mutually exclusive with input.snapshot
      service: eks
      accelerator: h100
      intent: training
      os: ubuntu
      platform: kubeflow
      nodes: 2
    profile: ""                      # optional name=value; empty uses the declared default
    # configuration:                 # typed desired-state inputs (not matched on)
    #   slurm:                       # only valid when the resolved recipe platform is slurm
    #     accounting:
    #       mode: disabled           # disabled | customer-managed | aicr-provided
    #   gke:                         # only valid for the h100 gke-cos kubeflow training leaf
    #     tcpxoInterfaces:           # required when that leaf ships torch-distributed-tcpxo;
    #       - {interfaceName: eth1, network: <cluster>-gpu-nic-0}  # ordered eth1..eth8 (8 entries);
    #                                                          # names from the cluster provisioner
    # input:
    #   snapshot: snapshot.yaml      # derive criteria from a snapshot instead
    output:
      path: recipe.yaml
      format: yaml                   # yaml | json | table
    data: ""                         # optional data-overlay dir/archive
    criteriaStrict: false            # reject criteria outside the embedded catalog
  bundle:
    input:
      recipe: recipe.yaml            # must match recipe.output.path when both set
    output:
      target: ./bundles              # local dir or oci:// URI
      imageRefs: ""                  # external digest file; OCI output only
    deployment:
      deployer: helmfile             # helm | helmfile | argocd | argocd-helm | flux
      repo: ""
      set: []                        # value overrides, "key:path=value"
      dynamic: []
      vendorCharts: false
      appName: ""                    # argocd parent Application name override
    scheduling:
      systemNodeSelector: {}
      systemNodeTolerations: []
      acceleratedNodeSelector:
        nodeGroup: gpu-worker
      acceleratedNodeTolerations:
        - nvidia.com/gpu=present:NoSchedule
      draEvictionNodeLabel: nvidia.com/dra-kubelet-plugin=true
      workloadGate: ""
      workloadSelector: {}
      nodes: 2
      storageClass: ""
      sharedStorageClass: ""          # RWX class for opt-in shared filesystems
    attestation:
      enabled: false
      certificateIdentityRegexp: ""
      oidcDeviceFlow: false
      fulcioURL: ""                  # private Sigstore overrides; empty = public good
      rekorURL: ""
      signingKey: ""                 # KMS key ref (awskms:// | gcpkms:// | ...); empty = keyless OIDC
    registry:                        # OCI transport for oci:// push
      insecureTLS: false
      plainHTTP: false
  validate:
    input:
      recipe: recipe.yaml
      snapshot: snapshot.yaml
    agent:                           # same nil-vs-empty semantics as snapshot.agent
      namespace: aicr-validation
      image: ""
      imagePullSecrets: []
      # Optional; omitted here so the defaults apply (both aicr-validate).
      # jobName is always a prefix with the run ID appended;
      # serviceAccountName is exact-if-exists — an existing ServiceAccount
      # of exactly that name is used verbatim and the run creates no RBAC.
      nodeSelector:
        nodeGroup: gpu-worker
      tolerations:
        - nvidia.com/gpu=present:NoSchedule
      requireGpu: false
    execution:
      phases: [deployment, conformance, performance]
      skipChecks: []                 # checks this run cannot satisfy; empty = run them all
      failOnError: true              # tri-state; absent = CLI default (true)
      failFast: false
      noCluster: false
      noCleanup: false
      timeout: 5m                    # aicr validate: live-capture agent Job only; inert here (input.snapshot is set)
    evidence:
      cncf:                          # CNCF AI Conformance markdown
        dir: ./evidence
        cncfSubmission: false        # requires dir
        features: []                 # empty = all features
      attestation:                   # recipe-evidence bundle (ADR-007)
        out: ./attestation           # directory; setting this enables the path
        bom: ""
        push: ""                     # OCI ref to push the signed bundle
        plainHTTP: false
        insecureTLS: false
  verify:                            # consumer side: policy for `aicr verify`
    policy:                          # assertions checked after verification
      minTrustLevel: verified        # unknown | unverified | attested | verified | max
      requireCreator: ci@myorg.example.com
      cliVersionConstraint: ">= 0.16.0"
    trust:                           # material verification runs against
      certificateIdentityRegexp: ""  # when set, must BEGIN with https://github.com/NVIDIA/aicr/
      key: ""                        # KMS URI or local PEM public-key path
      trustRoot: ""                  # private Sigstore trusted_root.json

Field Reference

spec.snapshot

Inputs to aicr snapshot. There is no input section — the snapshot is produced from the live cluster.

FieldTypeNotes
output.pathstringOutput file path (same as -o)
output.formatstringyaml | json | table
output.templatestringOptional Go template path
agent.*objectIn-cluster capture Job pod: namespace, image, imagePullSecrets, jobName, serviceAccountName, nodeSelector, tolerations, requireGpu, runtimeClassName (mutually exclusive with requireGpu), os, requests, limits. Mirrors spec.validate.agent so one file pins matching placement for both
agent.jobNamestringOptional prefix, never an exact name — the run ID is always appended (<prefix>-<run-id>). Omit it unless you need a custom prefix (default: aicr)
agent.serviceAccountNamestringOptional and exact-if-exists, so unlike jobName it is not always a prefix. When a ServiceAccount of exactly this name already exists in agent.namespace, the pod runs as it verbatim, the run creates and deletes no ServiceAccount, Role, RoleBinding, ClusterRole, or ClusterRoleBinding, and per-run permission isolation is waived — concurrent runs share that identity’s persistent grants. When it does not exist, the value is a prefix, the run ID is appended (<prefix>-<run-id>), and the run owns a full run-scoped RBAC set it deletes at cleanup. Omitting the field is not the same as writing aicr into it: an omitted name is never probed, so the run always takes the run-scoped aicr-<run-id>. See Using an existing ServiceAccount
execution.timeoutduration stringe.g. 5m
execution.noCleanupboolKeep the capture Job after completion
execution.privilegedbool (tri-state)Set false for PSS-restricted namespaces
execution.maxNodesPerEntryint0 = unlimited topology entries

spec.recipe

Inputs to aicr recipe. criteria and input.snapshot are mutually exclusive — query by criteria or derive from a snapshot, not both.

FieldTypeNotes
criteria.service / .accelerator / .intent / .os / .platformstringSame names and values as the CLI flags
criteria.nodesintTarget GPU node count
profilestringOptional configuration profile selection in name=value form. Empty applies the resolved declaration’s default.
configuration.slurm.accounting.modestringSlurm accounting ownership: disabled (default) | customer-managed | aicr-provided; mirrors --slurm-accounting-mode. Only valid when the resolved recipe platform is slurm (whether from criteria.platform, a snapshot, or --platform) — an explicit mode (even disabled) on any other platform is rejected with INVALID_REQUEST
configuration.runtimeInventory.modestringRuntime AI inventory (k8s-aibom) selection: enabled | disabled; mirrors --runtime-inventory
configuration.gke.tcpxoInterfaces[]objectOrdered interfaceName/network pairs (eth1–eth8) for the torch-distributed-tcpxo runtime; mirrors --gke-tcpxo-interfaces
input.snapshotstringSnapshot path to derive the recipe from
output.pathstringRecipe output path
output.formatstringyaml | json | table
datastringData-overlay directory/archive (same as --data)
criteriaStrictbool (tri-state)Reject criteria values outside the embedded catalog; mirrors --criteria-strict / AICR_CRITERIA_STRICT (any of the three enables it)

spec.bundle

Inputs to aicr bundle.

FieldTypeNotes
input.recipestringRecipe to bundle
output.targetstringLocal directory or oci:// URI
output.imageRefsstringOptional external image-reference output file for an OCI output.target only. Local output is rejected. Its parent must be an existing real directory, and the target must be outside and not aliased to the planned or completed bundle.
deployment.deployerstringDeployer choice (same values as --deployer)
deployment.repostringGitOps repo for repo-shaped deployers
deployment.set / .dynamic[]stringValue overrides, key:path=value
deployment.vendorChartsboolVendor charts into the bundle
deployment.appNamestringArgo CD parent Application name override (multi-bundle installs sharing a namespace)
scheduling.*objectsystemNodeSelector/Tolerations, acceleratedNodeSelector/Tolerations, draEvictionNodeLabel, workloadGate, workloadSelector, nodes, storageClass, sharedStorageClass. Selectors are YAML maps; tolerations use the CLI’s key=value:effect strings. draEvictionNodeLabel accepts one key=value label and has no default: setting it opts in to DRA eviction coordination, which AICR applies only when DRA and GPU Operator are both enabled.
attestation.enabledboolEnable bundle attestation (signing); keyless OIDC by default, KMS-backed when signingKey is set
attestation.certificateIdentityRegexpstringExpected signer identity
attestation.oidcDeviceFlowboolDevice-code flow for headless signing
attestation.fulcioURL / .rekorURLstringPrivate Sigstore endpoints; empty = public-good defaults
attestation.signingKeystringKMS key reference for key-based signing (awskms:// | gcpkms:// | azurekms:// | hashivault://); empty = keyless OIDC. Mutually exclusive with the keyless-only inputs (oidcDeviceFlow, fulcioURL, --identity-token)
registry.insecureTLS / .plainHTTPboolOCI transport options for push

When output.imageRefs is set, AICR writes the published OCI digest through a mode-0600 temporary file and an anchored same-directory rename. The target may be absent or an existing retained regular file; directories, symlinks, other non-regular files, and bundle aliases are rejected. The final validation and rename are ordered but are not one atomic identity-conditioned filesystem operation, so no other process should mutate the target directory while the bundle command runs.

spec.validate

Inputs to aicr validate.

FieldTypeNotes
input.recipe / .snapshotstringRecipe + snapshot to validate
agent.*objectThe live snapshot-capture Job pod aicr validate deploys when input.snapshot is empty; same fields and nil-vs-empty semantics as spec.snapshot.agent (minus runtimeClassName/os/requests/limits). Neither jobName nor serviceAccountName names the validator Jobs, whose RBAC is always run-scoped
agent.jobNamestringOptional prefix, never an exact name — the run ID is always appended (<prefix>-<run-id>). Omit it to take the default (aicr-validate)
agent.serviceAccountNamestringOptional and exact-if-exists, with the same two branches as spec.snapshot.agent.serviceAccountName: an existing ServiceAccount of exactly this name in agent.namespace is used verbatim, the run creates and deletes no RBAC, and per-run permission isolation is waived — concurrent runs share that identity’s persistent grants; when it does not exist the value is a prefix with the run ID appended (<prefix>-<run-id>) and the run owns a full run-scoped RBAC set. Omitting the field is not the same as writing aicr-validate into it: an omitted name is never probed, so the run always takes the run-scoped aicr-validate-<run-id>. See Using an existing ServiceAccount
execution.phases[]stringe.g. [deployment, conformance, performance]
execution.skipChecks[]stringChecks to withhold from every phase that runs, one level below phases. For a caller that cannot satisfy a check the recipe declares, e.g. a lane deploying a subset of the recipe. Each named check is reported as skipped, not dropped, so the CTRF report and the recipe-evidence bundle still account for it. Rejected before any validation resource is created when a name matches no check, when the list would leave a requested phase with nothing to run, or when it is set alongside evidence.cncf.dir (that renderer omits skipped checks, so a submission would silently lose the requirement). A cm:// recipe is read from the cluster first, so that form contacts the API server before the list is judged. Mirrors --skip-check
execution.failOnErrorbool (tri-state)Absent = CLI default (true); explicit false opts out
execution.failFastbool (tri-state)Stop after the first failed phase
execution.noClusterboolTest mode: no cluster access, constraints evaluated inline
execution.noCleanupboolKeep validation Jobs after completion
execution.timeoutduration stringaicr validate: timeout for the live snapshot-capture agent Job (--timeout, default 5m); inert when input.snapshot is set, and validator Jobs keep their per-check timeouts. Go SDK: Config.ValidateSettings().Timeout carries the same value for WithValidationTimeout, which caps the whole ValidateState run (see Go library)
evidence.cncf.dirstringCNCF AI Conformance evidence directory (--evidence-dir)
evidence.cncf.cncfSubmissionbool (tri-state)Emit submission layout; requires dir
evidence.cncf.features[]stringEmpty = all features; honored only with cncfSubmission
evidence.attestation.outstringOutput directory for the recipe-evidence bundle (predicateType v3; same as --emit-attestation), which receives summary-bundle/ and pointer.yaml — setting it enables the attestation path
evidence.attestation.bom / .pushstringBOM input; OCI ref for the signed bundle push
evidence.attestation.plainHTTP / .insecureTLSbool (tri-state)Push transport options

spec.verify

Verification policy for aicr verify, the one consumer-side section. The two sub-sections mirror how the command consumes them: policy holds assertions checked after verification runs, trust holds the material it verifies against.

FieldTypeNotes
policy.minTrustLevelstringunknown | unverified | attested | verified, or max (the CLI default) to auto-detect the highest level the bundle can reach
policy.requireCreatorstringPins the OIDC identity in the bundle attestation’s signing certificate
policy.cliVersionConstraintstringConstrains the aicr version in the attestation predicate; supports >=, >, <=, <, ==, !=, and a bare version means >=
trust.certificateIdentityRegexpstringCertificate identity pattern for binary attestation verification; must begin with https://github.com/NVIDIA/aicr/ (leading ^ allowed) and must not use top-level alternation, so it stays confined to the repository
trust.keystringKMS key URI (awskms:// | gcpkms:// | azurekms:// | hashivault://) or local PEM public-key path; the verify counterpart to spec.bundle.attestation.signingKey
trust.trustRootstringPath to a private Sigstore trusted_root.json, additive to the built-in public-good root

policy.minTrustLevel sets operator policy, not an org-enforced guardrail. A committed value lowers the effective floor as readily as it raises it (unknown makes the trust check a no-op, since every level meets it), so treat it as a reviewable choice rather than a control that cannot be relaxed.

A lowered floor admits any bundle whose actual trust level reaches it, which is broader than unsigned bundles. It also covers chains that legitimately degraded: an attested bundle whose binary attestation is absent, or one carrying external --data, both report attested against a verified maximum, so the default max rejects them while a lowered floor does not.

What no policy value can wave through: checksum failures, and attestations that are present but fail verification. Those are rejected regardless of the floor.

aicr verify logs the floor at INFO when config is what supplies it: that is, when --min-trust-level is absent and the configured value is anything other than max. An explicit flag takes precedence instead, and that override is logged separately by the flag-precedence path.

Every field is a durable, non-secret reference or policy value; no private key material is part of the schema. Three aicr verify flags are deliberately excluded: the bundle directory (a positional argument), --format (presentation, not policy), and --insecure-ignore-tlog, which weakens the trust floor and so stays command-line-only rather than something a committed file can silently enable. It still composes with a config-supplied trust.key.

Cross-Section Rules

  • At least one of the five spec.* sections must be present.
  • spec.recipe.criteria and spec.recipe.input.snapshot are mutually exclusive.
  • When both spec.recipe.output.path and spec.bundle.input.recipe are set, they must reference the same file (compared after filepath.Clean; mixing absolute and relative forms is rejected). Mismatched paths in a workflow file are almost always a typo, so the loader fails up-front.
  • Enum-valued fields (criteria.*, output formats, phases) are validated with the same parsers the CLI flags use, so error messages match the CLI’s.

See Also

  • CLI Reference — per-command flags and config-file-mode examples
  • Validation — validation phases and evidence workflow
  • Bundling — deployers and bundle outputs