> This page is for AI Cluster Runtime, version Latest · v1.0.0 (default). > For other versions, use one of these documentation indexes: > - Latest · v1.0.0 (default): https://docs.nvidia.com/aicr/latest/llms.txt > - v1.0.0: https://docs.nvidia.com/aicr/v1.0.0/llms.txt > - v0.22.0: https://docs.nvidia.com/aicr/v0.22.0/llms.txt > - v0.21.1: https://docs.nvidia.com/aicr/v0.21.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/aicr/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aicr/_mcp/server. # Artifact Verification A consumer-facing guide to verifying the artifacts `aicr` produces (deployment bundles and recipe-evidence bundles) across public-trust, KMS-key, and offline deployment shapes. For the exhaustive per-flag reference, see the [CLI reference](/aicr/user-guide/cli-reference): [`aicr verify`](/aicr/user-guide/cli-reference#aicr-verify), [`aicr evidence verify`](/aicr/user-guide/cli-reference#aicr-evidence-verify), and [`aicr trust update`](/aicr/user-guide/cli-reference#aicr-trust-update). ## What Can Be Verified | Artifact | Produced by | Verified by | |----------|-------------|-------------| | Deployment bundle | `aicr bundle` | [`aicr verify`](/aicr/user-guide/cli-reference#aicr-verify) | | Recipe-evidence bundle | `aicr validate --emit-attestation` | [`aicr evidence verify`](/aicr/user-guide/cli-reference#aicr-evidence-verify) | | Embedded recipe catalog | shipped in the `aicr` binary | [`aicr recipe verify-catalog`](/aicr/user-guide/cli-reference#aicr-recipe-verify-catalog) | This guide focuses on the first two. Catalog verification is a single self-contained command; see its [CLI reference entry](/aicr/user-guide/cli-reference#aicr-recipe-verify-catalog). Bundles are unsigned by default; attestation is opt-in via `aicr bundle --attest`. An unsigned bundle can still have its closed-world inventory verified, but it cannot reach the higher trust levels described below. AICR treats a deployment bundle as a closed-world inventory. `checksums.txt` contains one SHA256 entry for every regular payload file, including `recipe.yaml` when present. Verification derives the required directories from those paths and rejects every additional file or directory, symlink, and other non-regular filesystem object. Only `checksums.txt`, `attestation/bundle-attestation.sigstore.json`, and `attestation/aicr-attestation.sigstore.json` may exist outside the manifest; they remain part of the verified inventory. Inventory validation accepts valid manifest entries in any order, while AICR generates them sorted by canonical slash-relative path. AICR ZIP and CLI OCI publication revalidate a private snapshot and publish only that inventory. Legacy bundles with incomplete manifests report `unknown` trust and must be regenerated. ## Released Container Image Metadata The artifacts above are the ones *you* produce with `aicr`. The container images NVIDIA publishes carry their own metadata, and all of it is retrievable from the registry as signed OCI referrers on the image: SLSA build provenance on the multi-platform index, plus a CycloneDX SBOM and an OpenVEX document on each platform's own manifest digest, recording the CVEs AICR has triaged as not affected. One command per kind: ```shell # Extract under pipefail, and only write a predicate file once the whole # pipeline has succeeded. `jq` exits 0 on empty input, so without pipefail a # failed `cosign verify-attestation` would still end the pipeline with status 0, # and a plain `>` redirect would leave a zero-length file that reads as a # successful extraction. set -o pipefail export IMAGE="ghcr.io/nvidia/aicr" export TAG=$(curl -s https://api.github.com/repos/NVIDIA/aicr/releases/latest | jq -r '.tag_name') export DIGEST=$(crane digest "${IMAGE}:${TAG}") # Resolve the platform digest from the pinned index, not from the tag: a tag # repointed between the two lookups would pair this SBOM and VEX with a # different release's index. export DIGEST_AMD64=$(crane digest --platform linux/amd64 "${IMAGE}@${DIGEST}") export AICR_ISSUER="https://token.actions.githubusercontent.com" export AICR_SIGNER="https://github.com/NVIDIA/aicr/.github/workflows/attest-images.yaml@refs/tags/${TAG}" # Build provenance (attached to the index digest and to each platform manifest; # any of the three verifies, and the index copy is the one an admission policy can reach) gh attestation verify "oci://${IMAGE}@${DIGEST}" --repo NVIDIA/aicr \ --signer-workflow NVIDIA/aicr/.github/workflows/attest-images.yaml \ --source-ref "refs/tags/${TAG}" \ --bundle-from-oci # CycloneDX SBOM (attached to the per-platform manifest digest) sbom=$(cosign verify-attestation --type cyclonedx \ --certificate-oidc-issuer "${AICR_ISSUER}" \ --certificate-identity "${AICR_SIGNER}" \ "${IMAGE}@${DIGEST_AMD64}" | jq -r '.payload' | base64 -d | jq '.predicate') \ && printf '%s\n' "${sbom}" > sbom.cdx.json # OpenVEX triage document (attached to the same per-platform manifest digest) openvex=$(cosign verify-attestation --type openvex \ --certificate-oidc-issuer "${AICR_ISSUER}" \ --certificate-identity "${AICR_SIGNER}" \ "${IMAGE}@${DIGEST_AMD64}" | jq -r '.payload' | base64 -d | jq '.predicate') \ && printf '%s\n' "${openvex}" > aicr-openvex.json ``` `--bundle-from-oci` makes `gh attestation verify` read the provenance from the registry referrer. Without it the command fetches from GitHub's attestations API, which is a different discovery path and would still succeed if the referrer push had silently failed. `--source-ref` and the exact `--certificate-identity` bind the result to release `${TAG}` rather than to any NVIDIA/aicr release. The SBOM and the VEX are attested to the platform manifest, not to the index, because each describes exactly one root filesystem, and because a consumer that has resolved `linux/amd64` enumerates referrers there. Repeat both commands against `linux/arm64` to retrieve that platform's pair. The two documents are in different formats on purpose: a referrer descriptor has no name field, so publishing the VEX as OpenVEX rather than as a second CycloneDX document is what lets you tell them apart in a listing without downloading both. Pass the VEX document to your scanner so it suppresses the same findings AICR's own release scan suppresses, instead of re-reporting CVEs that have already been analyzed as unreachable: ```shell grype "${IMAGE}@${DIGEST_AMD64}" --vex aicr-openvex.json --only-fixed --fail-on high trivy image --vex aicr-openvex.json "${IMAGE}@${DIGEST_AMD64}" ``` Each statement names one product, `pkg:oci/@sha256:`, so it binds to the manifest it was triaged against; scan the same digest you retrieved it from. Each also carries a `justification` and an `impact_statement` explaining why the vulnerable code is not reachable. Read those before adopting a suppression: the analysis is specific to how AICR invokes the affected component. An empty `statements` array is a valid result, not a failure: it means AICR has triaged no CVE for that image. Most released images are in that state at any given time; which ones is a property of the release you are verifying, so read the document you retrieved rather than assuming. **An empty VEX means "no exceptions asserted", not "no vulnerabilities."** It says what AICR has claimed, not what a scanner would find, so do not read one as a clean bill of health, and do not read the absence of a statement about a CVE as a claim that the image is unaffected by it. Every released image carries a document so that "no exceptions asserted" stays distinguishable from evidence that never reached the registry. The two `cosign verify-attestation` commands above need Cosign v3.0.1 or newer; `gh attestation verify` ships with the GitHub CLI and does not use Cosign, so it carries no such floor. For the full walkthrough, including the remaining release images and the signed binary SBOMs, see [Supply Chain Verification](/aicr/integrator-guide/supply-chain-verification#unified-metadata-retrieval). ## Trust Levels `aicr verify` computes a trust level for a bundle. The levels are ordered; each one subsumes the guarantees of the levels beneath it. | Level | Name | What it guarantees | |-------|------|--------------------| | 4 | `verified` | Closed-world inventory valid, bundle attestation verified, binary attestation verified with identity pinned to NVIDIA CI, and no external data | | 3 | `attested` | Closed-world inventory and bundle attestation valid, but the chain is incomplete — the binary attestation is missing or external `--data` was used | | 2 | `unverified` | Closed-world inventory valid, but no attestation files exist (the bundle was created without `--attest`) | | 1 | `unknown` | Inventory is missing, incomplete, or invalid; or an attestation (bundle or binary) is present but *fails* verification — a present binary attestation that does not verify is a hard failure, not a degraded `attested` | The ordering matters for enforcement: `verified` > `attested` > `unverified` > `unknown`. A bundle that uses external data can never exceed `attested`, and a bundle created without `--attest` can never exceed `unverified`, regardless of anything else. This is why `--min-trust-level max` (the default) auto-detects the *highest level a given bundle could achieve* and verifies against that, rather than failing a deliberately unsigned bundle. ## Public-Trust Bundle Verification The default path. Verification is offline and makes no network calls (the one exception, a KMS URI passed to `--key`, is covered below). ```shell # Verify a bundle, auto-detecting and enforcing its maximum achievable trust level. aicr verify ./my-bundle ``` Under the hood this runs three checks: 1. **Closed-world inventory**: every regular payload file, including `recipe.yaml` when present, must have a matching entry in `checksums.txt`; every additional filesystem entry is rejected except the three allowed inventory metadata files. 2. **Bundle attestation**: the bundle's signature is verified against the Sigstore trusted root. 3. **Binary attestation**: the provenance chain is verified with identity pinned to NVIDIA CI. Manifest entry order does not affect inventory validation. AICR nevertheless generates `checksums.txt` in canonical slash-relative path order. Reordering an already signed manifest changes its bytes and invalidates the existing bundle attestation. You can also pin the bundle's creator identity or the CLI version that produced it: ```shell # Require that a specific identity created the bundle. aicr verify ./my-bundle --require-creator jdoe@company.com # Require a minimum aicr CLI version (recorded in the attestation predicate). aicr verify ./my-bundle --cli-version-constraint ">= 0.8.0" ``` ## Minimum Trust Level Enforcement By default `aicr verify` uses `--min-trust-level max`, which resolves to the highest level the bundle could achieve and fails if verification falls short. To require an explicit floor regardless of the bundle's contents, name a level: ```shell # Fail unless the bundle reaches full `verified` trust. aicr verify ./my-bundle --min-trust-level verified # Accept `attested` or higher (e.g. when external --data is expected). aicr verify ./my-bundle --min-trust-level attested ``` Valid values are `verified`, `attested`, `unverified`, `unknown`, and `max`. A bundle whose computed level is below the requested floor exits non-zero. An incomplete legacy manifest cannot establish a closed-world inventory, so it reports `unknown` and the bundle must be regenerated. ## KMS-Key Verification Some environments cannot use keyless OIDC signing, so bundles are signed with a KMS key via `aicr bundle --attest --signing-key `. Verify those bundles with `--key`, supplying the same KMS URI used to sign. Supported schemes are `awskms://`, `gcpkms://`, `azurekms://`, and `hashivault://`. ```shell # Sign with a KMS key, then verify it with the same key. aicr bundle -r recipe.yaml --attest \ --signing-key gcpkms://projects/p/locations/l/keyRings/r/cryptoKeys/k \ -o ./bundles aicr verify ./bundles/ \ --key gcpkms://projects/p/locations/l/keyRings/r/cryptoKeys/k ``` Resolving a KMS URI makes network calls to the KMS provider to fetch the public key, so credentials for that provider must be available in the environment. ## Local PEM Key Verification To verify without granting KMS access, export the public key once and verify against the local PEM file; this part makes no provider calls: ```shell # Export the public key once (requires KMS access at export time). cosign public-key --key gcpkms://projects/p/locations/l/keyRings/r/cryptoKeys/k > bundle-signer.pub # Verify anywhere afterward, with no KMS access needed. aicr verify ./bundles/ --key ./bundle-signer.pub ``` A local PEM key is read from disk only. The bundle's Rekor transparency-log entry is also checked by default, but its inclusion proof is embedded in the bundle and the trusted root falls back to the embedded copy on a cache miss — so verification runs fully offline without `aicr trust update`. ## Privately-Signed Bundle Verification Air-gapped and private-deployment sites often run their own Sigstore stack (a self-hosted Fulcio CA and Rekor log, for example from sigstore/scaffolding) rather than the public-good infrastructure. `aicr bundle --attest` signs against that stack with `--fulcio-url` / `--rekor-url`; `aicr verify --trust-root` is the verify counterpart. Point it at the `trusted_root.json` your private Sigstore stack emits: ```shell # Verify a bundle signed against a private Fulcio/Rekor. aicr verify ./my-bundle --trust-root ./trusted_root.json ``` `--trust-root` is **additive**: the supplied root is unioned with AICR's built-in public-good root, not substituted for it. So a single command verifies both org-signed bundles (against the private root) and NVIDIA-signed bundles (against the public-good root); you do not need to switch flags per artifact. The flag supplies trust anchors only; it does not by itself require a bundle to be privately signed. To pin a specific signer, keep using `--require-creator` (and, for the binary attestation's identity, `--certificate-identity-regexp`): ```shell # Verify against the org root and require a specific signer identity. aicr verify ./my-bundle \ --trust-root ./trusted_root.json \ --require-creator ci@myorg.example.com ``` `--trust-root` composes with `--key`, so a KMS- or PEM-key-signed bundle whose signature was logged to a private Rekor verifies with both flags together: ```shell aicr verify ./my-bundle \ --trust-root ./trusted_root.json \ --key ./bundle-signer.pub ``` Only the **bundle attestation** consults the private root. The **binary attestation** is always produced by NVIDIA's public CI and continues to verify against the public-good root, regardless of `--trust-root`. Errors with the supplied `trusted_root.json` (missing, unreadable, oversized, or malformed) are reported as invalid-request failures. ## Offline and Air-Gapped Considerations `aicr verify` is offline by default and does not call out to Sigstore at verify time; the Rekor inclusion proof is embedded in the bundle, so no live Rekor call is made. The check needs the Sigstore *trusted root*, but verification forces the local cache (`WithForceCache`) and falls back to the **embedded** trusted root on a cache miss — it does not fetch on the verify path, so offline verification works out of the box. `aicr trust update` refreshes the cached trust material (it contacts the TUF CDN once) but is **not** required for offline verification. Two further shapes, both reflected in the current CLI reference: - **Fully transparency-log-free verification**: a bundle signed with `aicr bundle --attest --signing-key --tlog-upload=false` carries no Rekor entry at all. Verify it with `aicr verify ./my-bundle --key ./bundle-signer.pub --insecure-ignore-tlog`, which drops the transparency-log requirement entirely. The flag requires `--key`, because the air-gapped path is key-based rather than keyless, and it is named "insecure" because with no transparency log there is no trusted timestamp proving when the signature was made. It does not affect the binary attestation, which always requires a transparency log. - **Private Sigstore verification**: `aicr bundle --attest` can redirect *signing* to a private Fulcio/Rekor with `--fulcio-url` / `--rekor-url`, and `aicr verify --trust-root` verifies the resulting bundles against that infrastructure's `trusted_root.json`. See [Privately-Signed Bundle Verification](#privately-signed-bundle-verification). ## Recipe Evidence Verification Recipe-evidence bundles, produced by `aicr validate --emit-attestation`, are verified with `aicr evidence verify`. When the bundle carries a signature, the command verifies it against the Sigstore trusted root, recomputes every manifest-listed payload file's sha256 against `manifest.json`, and surfaces the predicate's fingerprint, phase counts, and BOM info. Bundles are **minimized by default**: the published `snapshot.yaml` keeps only an allowlisted set of fields and the CTRF reports omit per-test stdout/message, keeping sensitive operational detail (node names, provider instance IDs, the node label/taint set, OS tuning, raw container logs) out of the published artifact. A small allowlisted set of structured, low-cardinality per-test outcome fields (the CTRF `extra` object — coverage counts and skip-reason codes, never node names or IPs) is *preserved* so a signed bundle still shows e.g. a reduced-node pass; see [Structured evidence that survives redaction](/aicr/contributor-guide/validators#structured-evidence-that-survives-redaction). The predicate records the applied policy in a `redaction` block, which `aicr evidence verify` surfaces. Minimal bundles self-verify exactly like full ones — the digests cover whatever bytes shipped. Pass `--full` to `aicr validate --emit-attestation` to publish the raw payloads instead. The positional argument is auto-detected. Prefer the committed **pointer file**, which pins the bundle by digest: ```shell # Verify a pointer a contributor committed alongside a recipe change (preferred). aicr evidence verify recipes/evidence///.yaml # Verify a pushed OCI bundle directly, pinned by digest. aicr evidence verify ghcr.io/myorg/aicr-evidence@sha256:abc... # Verify a local bundle directory (self-debug before push). aicr evidence verify ./out/summary-bundle ``` To require a specific signer, pin the OIDC issuer and identity: ```shell aicr evidence verify recipes/evidence///.yaml \ --expected-issuer https://token.actions.githubusercontent.com \ --expected-identity-regexp '^https://github\.com/myorg/myrepo/\.github/workflows/release\.yaml@refs/tags/.+$' ``` ## Per-Source Pointer Layout and the Signer Allowlist Committed evidence pointers are **per-source** so two parties can attest to the same recipe without overwriting each other. The on-disk layout is add-only and immutable (ADR-007; issue [#1347](https://github.com/NVIDIA/aicr/issues/1347)): ``` recipes/evidence///.yaml # one immutable pointer per run recipes/evidence/allowlist.yaml # maintained signer allowlist ``` Each file is a single-attestation V1 pointer — the same schema that `aicr evidence verify` already consumes. The `` segment is a stable slug derived from the signer OIDC identity recorded in the pointer — the first 32 hex characters (128 bits) of `sha256(issuer + "\n" + identity)`. Because the slug is computed from the signer fields rather than chosen freely, the `Evidence Pointer Contract` CI job recomputes the slug from each pointer's own signer block and rejects any file that does not live under the directory its signer hashes to. > **Where cryptographic trust enters (#1535).** The on-disk > pointer-contract merge gate is *structural*: it derives the slug from the > signer fields the pointer declares and checks path ownership + allowlist > membership, but does not itself verify a signature. Cryptographic trust is > enforced **after merge, at ingest** (`evidence-ingest.yaml`), which verifies > the signature pinned to the claimed signer and cross-checks the certificate > before any result is counted — so a pointer that lied about its signer passes > the gate but fails ingest. See > [#1535](https://github.com/NVIDIA/aicr/issues/1535) and ADR-007. Consumers discover a recipe's evidence by glob — `recipes/evidence//*/*.yaml` — and aggregate across sources; nothing is ever modified in place. The **allowlist** (`recipes/evidence/allowlist.yaml`) is the trust root. It pins, per class — `first-party`, `community`, `partner` — the signers that may contribute corroborating evidence. Community and partner entries are keyed by the one-way `source` slug only — the cleartext identity (e.g. a personal email) is **never** stored in the *allowlist*; an optional non-PII `label` (e.g. a GitHub handle) is for display and may be omitted. (The committed pointer and Rekor still carry `signer.identity` — see the note below — so a contributor is not anonymous.) First-party entries pin a tightly-bounded anchored `identityPattern` whose issuer/org/repo/workflow segment is literal (a wildcard is permitted only in the ref) — these are CI workflow URLs, not personal identities. The classes are disjoint and no two entries overlap, so a verified signer classifies exactly one way. A verified signer that is **not** listed is admitted as *reported* only — it never counts toward corroboration. (Only the **allowlist** is slug-only — its loader rejects a stray `identity:` field. Committed **pointers** (`recipes/evidence///*.yaml`) do carry `signer.identity`, and keyless Sigstore also records it in the public Rekor log, so the signer identity is public regardless. Sign with a non-personal identity — a GitHub `users.noreply.github.com` address or CI's ambient OIDC — not a personal email.) First-party AICR CI (the UAT workflows) signs with the GitHub Actions OIDC issuer and **ingests evidence directly** — it does not commit a per-run pointer, which would churn `main` nightly. Committed per-source pointers are therefore the community/partner channel. ### Contributing evidence (community / partner) 1. Run `aicr validate --emit-attestation` and push the signed bundle to a registry you control. The command prints a `copyTo` hint with the exact per-source destination path. 2. Open a PR that adds **your own** pointer file under `recipes/evidence///` — alongside, never replacing, any other party's — plus a slug entry (`source:` + `issuer:`, optional `label:`) for your signer in the correct class of `recipes/evidence/allowlist.yaml`. The `` slug is printed in the `copyTo` hint; use that same value as `source:`. 3. Maintainer review (enforced by `CODEOWNERS`) is the trust gate; the `Evidence Pointer Contract` job verifies the path-ownership and allowlist invariants before merge. A tag-only OCI reference is refused by default because tags are registry-rewritable; pass the pointer file (which carries the digest) or a digest-pinned ref instead. See the [`aicr evidence verify`](/aicr/user-guide/cli-reference#aicr-evidence-verify) reference for the full input-detection rules and the `--allow-unpinned-tag` escape hatch. ## JSON Output for CI Both verifiers emit machine-readable JSON for pipeline gating. ```shell # Bundle verification as JSON. aicr verify ./my-bundle --format json # Evidence verification as JSON to a file. aicr evidence verify recipes/evidence///.yaml --format json -o result.json ``` `aicr evidence verify` exits `0` when every check passes, `2` when the bundle is invalid or recorded validator results show failures, `5` when verification could not complete because the bundle was not readable (a dead mount or an unreachable registry), and `9` when the run was aborted by the operator. The last two are deliberately distinct process codes: nothing was proven about the bundle in either case, so a gate must not reject the artifact. Retry the `5` case; a `9` means someone stopped the run on purpose. The JSON output's `exit` field further distinguishes recorded phase failures (`1`) from an invalid bundle (`2`), so a shell consumer can branch on it. Write the JSON to a file and read the `exit` field from it rather than piping `aicr evidence verify` straight into `jq`: under `set -o pipefail` (common in CI) the verifier's non-zero exit would otherwise propagate through the pipeline and abort the script before the `case` runs. The `|| true` keeps that exit from tripping `set -e`: ```shell rm -f result.json aicr evidence verify recipes/evidence///.yaml --format json -o result.json || true if [ ! -s result.json ]; then echo "verifier wrote no result — treat as a failure"; exit 1 fi case "$(jq '.exit' result.json)" in 0) echo "evidence valid" ;; 1) echo "validator phases failed"; exit 1 ;; 2) echo "bundle invalid"; exit 1 ;; 3) echo "no verdict reached (infrastructure fault or aborted) — retry, do not reject"; exit 3 ;; *) echo "unrecognized verdict — treat as a failure"; exit 1 ;; esac ``` ## Monitoring Your Signing Identity Verification is only half the picture. It proves that an artifact you hold came from the identity you expect, but it cannot tell you that somebody else signed something *as you*: an entry in the transparency log under your signing identity that you did not produce may indicate the identity was used without you. Keyless signing leaves no local trace of a signing event, so the log is the only place that misuse shows up. For what each signing mode records and how to watch for it, see [Monitoring Your Signing Identity](/aicr/integrator-guide/supply-chain-verification#monitoring-your-signing-identity). ## Troubleshooting Common Failures **Certificate chain errors / stale trusted root.** Sigstore rotates signing keys a few times per year. If verification fails with certificate-chain errors, refresh the local trusted root: ```shell aicr trust update ``` **`--key` cannot reach the KMS provider.** A KMS URI requires provider credentials in the environment. If you only need to verify (not sign), export the public key once with `cosign public-key --key ` and pass the local PEM file to `--key` instead; see [Local PEM Key Verification](#local-pem-key-verification). **Tag-only OCI reference refused.** `aicr evidence verify` rejects tag-only references because tags are registry-rewritable. Pass the committed pointer file (which carries the `sha256:` digest) or a digest-pinned ref (`...@sha256:`). **Trust level lower than expected.** A bundle created without `--attest` caps at `unverified`; a bundle built with external `--data` caps at `attested`. If you require a stricter level, re-create the bundle with attestation and without external data, then verify with `--min-trust-level verified`. If an older bundle has an incomplete `checksums.txt`, regenerate it; listed-digest checks cannot establish closed-world trust. **Private-Sigstore-signed bundle won't verify.** A bundle signed against a private Fulcio/Rekor needs that infrastructure's trusted root. Pass it with `aicr verify --trust-root ./trusted_root.json`. See [Privately-Signed Bundle Verification](#privately-signed-bundle-verification).