Supply Chain Verification | NVIDIA AI Cluster Runtime

This guide is for integrators wiring AICR artifact verification into CI pipelines, clusters, and audit tooling. It collects the full command walkthroughs for verifying build provenance, SBOMs, and image/bundle attestations, plus admission-policy enforcement and offline/air-gapped verification.

For a quick trust overview and how to report a vulnerability, see the top-level SECURITY.md.

Prerequisites and Setup

Verification uses Cosign, the GitHub CLI (gh), crane (or docker), jq, and — for in-cluster enforcement — kubectl. Binary and bundle verification (aicr verify) need only the aicr binary.

Export the following variables once; the rest of this guide reuses them. Tags are mutable and can be repointed to a different image, so resolve the tag to an immutable @sha256: digest and verify against the digest.

$ # Latest release tag
$ export TAG=$(curl -s https://api.github.com/repos/NVIDIA/aicr/releases/latest | jq -r '.tag_name')
$ export VERSION=${TAG#v}  # strip leading 'v' for release filenames
$ 
$ # CLI image
$ export IMAGE="ghcr.io/nvidia/aicr"
$ export DIGEST=$(crane digest "${IMAGE}:${TAG}" 2>/dev/null || docker inspect "${IMAGE}:${TAG}" --format='{{index .RepoDigests 0}}' | cut -d'@' -f2)
$ export IMAGE_DIGEST="${IMAGE}@${DIGEST}"
$ export IMAGE_SBOM="$IMAGE:sha256-$(echo "$DIGEST" | cut -d: -f2).sbom"
$ 
$ # API server image
$ export IMAGE_API="ghcr.io/nvidia/aicrd"
$ export DIGEST_API=$(crane digest "${IMAGE_API}:${TAG}")
$ export IMAGE_API_DIGEST="${IMAGE_API}@${DIGEST_API}"

Authentication (if the registry requires it):

$ docker login ghcr.io

Verifying Build Provenance (SLSA)

AICR produces SLSA build provenance through GitHub Actions: builds are defined as code and provenance is service-generated (signed by GitHub’s OIDC-authenticated attestation service via actions/attest-build-provenance) rather than self-asserted, then logged to the public Rekor transparency log.

Note on the SLSA Build Level. GitHub’s attestation service yields Build Level 2 by default; Build Level 3 additionally requires build isolation via a dedicated reusable workflow. AICR currently attests from its release job using composite actions (not an isolated reusable workflow), so the exact level claimed elsewhere in the docs is being reconciled with the build architecture in #1536.

Method 1: GitHub CLI

$ # Verify provenance exists and is valid (using digest)
$ gh attestation verify oci://${IMAGE_DIGEST} --owner nvidia
$ 
$ # Output shows:
$ # ✓ Verification succeeded!
$ #
$ # Attestations:
$ #   • Build provenance (SLSA v1.0)
$ #   • SBOM (SPDX)

Method 2: Extract and inspect provenance

$ # Get full provenance data (using digest)
$ gh attestation verify oci://${IMAGE_DIGEST} \
>   --owner nvidia \
>   --format json | jq '.[] | select(.verificationResult.statement.predicateType | contains("slsa"))'
$ 
$ # Key fields in provenance:
$ # - buildDefinition.buildType: GitHub Actions workflow type
$ # - runDetails.builder.id: Workflow file and commit
$ # - buildDefinition.externalParameters.workflow: Workflow path and ref
$ # - buildDefinition.resolvedDependencies: Source code commit SHA
$ # - runDetails.metadata.invocationId: GitHub run ID

The signed certificate binds the artifact to its source repository, commit SHA, workflow, and run. A representative slice:

1 {
2   "verificationResult": {
3     "signature": {
4       "certificate": {
5         "subjectAlternativeName": "https://github.com/NVIDIA/aicr/.github/workflows/on-tag.yaml@refs/tags/v0.8.12",
6         "issuer": "https://token.actions.githubusercontent.com",
7         "githubWorkflowName": "on_tag",
8         "githubWorkflowRepository": "NVIDIA/aicr",
9         "githubWorkflowRef": "refs/tags/v0.8.12",
10         "sourceRepositoryURI": "https://github.com/NVIDIA/aicr",
11         "sourceRepositoryDigest": "ba6cbbe8b1a8fc8b72bb18454c10a3ba31d94a2e",
12         "runnerEnvironment": "github-hosted",
13         "runInvocationURI": "https://github.com/NVIDIA/aicr/actions/runs/20642050863/attempts/1"
14       }
15     }
16   }
17 }

Build process transparency

All AICR releases are built using GitHub Actions with full transparency:

Source Code — Public GitHub repository
Build Workflow — .github/workflows/on-tag.yaml (version controlled)
Build Logs — Public GitHub Actions run logs
Attestations — Signed and stored in the public transparency log (Rekor)
Artifacts — Published to GitHub Releases and GHCR

View build history:

$ # List all releases with attestations
$ gh api repos/NVIDIA/aicr/releases | \
>   jq -r '.[] | "\(.tag_name): \(.html_url)"'
$ 
$ # View specific build logs
$ gh run list --repo NVIDIA/aicr --workflow=on-tag.yaml
$ gh run view 20642050863 --repo NVIDIA/aicr --log

Verify in the transparency log (Rekor):

$ # Search Rekor for attestations
$ rekor-cli search --artifact ghcr.io/nvidia/aicr:v0.8.12
$ 
$ # Get entry details
$ rekor-cli get --uuid <entry-uuid>

Verifying the SBOM

AICR provides SBOMs in SPDX v2.3 JSON format: binary SBOMs embedded in CLI binaries (generated by GoReleaser) and container image SBOMs attached as Cosign attestations (generated by Syft/Anchore).

Binary SBOM (CLI)

$ # Detect OS and architecture
$ export OS=$(uname -s | tr '[:upper:]' '[:lower:]')
$ export ARCH=$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/')
$ 
$ # Download the versioned archive from GitHub releases and extract the binary.
$ # GoReleaser ships aicr_<version>_<os>_<arch>.tar.gz; ${VERSION} is the tag
$ # without its leading "v" (e.g. TAG=v0.8.12 -> VERSION=0.8.12).
$ curl -LO https://github.com/NVIDIA/aicr/releases/download/${TAG}/aicr_${VERSION}_${OS}_${ARCH}.tar.gz
$ tar -xzf aicr_${VERSION}_${OS}_${ARCH}.tar.gz
$ chmod +x aicr
$ 
$ # Download SBOM (separate file)
$ curl -LO https://github.com/NVIDIA/aicr/releases/download/${TAG}/aicr_${VERSION}_${OS}_${ARCH}.sbom.json
$ 
$ # View SBOM
$ cat aicr_${VERSION}_${OS}_${ARCH}.sbom.json

Container image SBOM

$ # Method 1: Using Cosign (extracts attestation) - uses digest to avoid warnings
$ cosign verify-attestation \
>   --type spdxjson \
>   --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>   --certificate-identity-regexp 'https://github.com/NVIDIA/aicr/.github/workflows/.*' \
>   ${IMAGE_API_DIGEST} | \
>   jq -r '.payload' | base64 -d | jq '.predicate' > sbom.json
$ 
$ # Method 2: Using GitHub CLI (shows all attestations)
$ gh attestation verify oci://${IMAGE_API_DIGEST} --owner nvidia --format json

SBOM format

Both binary and container SBOMs are SPDX v2.3 JSON. A representative package entry (the full document lists every Go module and its transitive dependencies, licenses, and package URLs):

1 {
2   "spdxVersion": "SPDX-2.3",
3   "dataLicense": "CC0-1.0",
4   "name": "aicr",
5   "creationInfo": {
6     "creators": ["Organization: Anchore, Inc", "Tool: syft-1.38.2"]
7   },
8   "packages": [
9     {
10       "name": "github.com/NVIDIA/aicr",
11       "versionInfo": "v0.8.12",
12       "externalRefs": [
13         {
14           "referenceType": "purl",
15           "referenceLocator": "pkg:golang/github.com/NVIDIA/aicr@v0.8.12"
16         }
17       ]
18     }
19   ]
20 }

SBOM use cases

$ # Vulnerability scanning — feed the SBOM to Grype, Anchore, or Snyk
$ grype sbom:./sbom.json
$ 
$ # License compliance — list declared licenses
$ jq -r '.packages[] | select(.licenseDeclared != "NOASSERTION") | "\(.name) \(.versionInfo) \(.licenseDeclared)"' sbom.json
$ 
$ # Dependency tracking — search for a specific component
$ jq '.packages[] | select(.name | contains("vulnerable-lib"))' sbom.json
$ 
$ # Audit trail — the SBOM timestamp proves when components were included
$ jq '.creationInfo.created' sbom.json

Verifying Image and Bundle Attestations

Container image attestations

Method 1: GitHub CLI (recommended)

$ # Verify using digest (preferred - no warnings)
$ gh attestation verify oci://${IMAGE_DIGEST} --owner nvidia
$ 
$ # Verify the aicrd image
$ gh attestation verify oci://${IMAGE_API_DIGEST} --owner nvidia
$ 
$ # Note: You can still use tags, but tools may show warnings about mutability
$ # gh attestation verify oci://ghcr.io/nvidia/aicr:${TAG} --owner nvidia

Method 2: Cosign (SBOM attestations)

$ # Verify SBOM attestation using digest (preferred - avoids warnings)
$ cosign verify-attestation \
>   --type spdxjson \
>   --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>   --certificate-identity-regexp 'https://github.com/NVIDIA/aicr/.github/workflows/.*' \
>   ${IMAGE_DIGEST}
$ 
$ # Extract and view the SBOM predicate
$ cosign verify-attestation \
>   --type spdxjson \
>   --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>   --certificate-identity-regexp 'https://github.com/NVIDIA/aicr/.github/workflows/.*' \
>   ${IMAGE_DIGEST} | jq -r '.payload' | base64 -d | jq '.predicate'

CLI binary attestation

CLI binary releases are attested with SLSA Build Provenance v1 using Cosign keyless signing via GitHub Actions OIDC. Each release archive (.tar.gz) contains the aicr binary and an aicr-attestation.sigstore.json Sigstore bundle. The attestation is logged to the public Rekor transparency log and can be verified offline.

$ cosign verify-blob-attestation \
>   --bundle aicr-attestation.sigstore.json \
>   --type https://slsa.dev/provenance/v1 \
>   --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>   --certificate-identity-regexp 'https://github.com/NVIDIA/aicr/.github/workflows/on-tag\.yaml@refs/tags/.*' \
>   aicr

The install script (./install) runs this verification automatically when Cosign is available. The Build Attested Binaries workflow (.github/workflows/build-attested.yaml) can be triggered manually from the Actions tab to produce attested binaries from any branch without cutting a release.

Bundle attestation

When aicr bundle runs with --attest, it signs the bundle using Sigstore keyless OIDC, binding the bundle creator’s identity to the bundle content (via checksums.txt) and the binary that produced it (via resolvedDependencies). Attestation is opt-in; bundles are unsigned by default. The bundle output includes bundle-attestation.sigstore.json (SLSA Build Provenance v1 for the bundle) and a copy of the binary’s aicr-attestation.sigstore.json (provenance chain).

$ aicr verify ./my-bundle

This verifies checksums against checksums.txt, the bundle attestation against the Sigstore trusted root, and the binary attestation provenance chain (identity pinned to NVIDIA CI). Enforce a minimum trust level:

$ aicr verify ./my-bundle --min-trust-level verified

For full CLI flag documentation, see the CLI Reference (aicr verify, aicr bundle --attest, aicr trust update). For a hands-on walkthrough, see the Bundle Attestation Demo.

Enforcing with Admission Policies

You can enforce provenance verification at deployment time with a Kubernetes admission controller. AICR’s images carry GitHub Artifact Attestations, which are Sigstore bundles — so the admission policy must verify the Sigstore bundle format (not the legacy Cosign signature format):

Kyverno — verify with type: SigstoreBundle; see Kyverno’s Verifying Sigstore Bundles guide.
Sigstore Policy Controller — Sigstore-bundle support requires v0.13.0+ and signatureFormat: bundle; see the Sigstore bundle format docs. Enforcement only runs in namespaces labeled policy.sigstore.dev/include=true.

Verify against AICR’s release identity:

issuer: https://token.actions.githubusercontent.com
subject: https://github.com/NVIDIA/aicr/.github/workflows/on-tag.yaml@refs/tags/* (the release workflow; narrow to the release pattern rather than trusting every workflow/ref)

Validated, cluster-tested copy-paste policies (Kyverno SigstoreBundle and Policy Controller v0.13+) are tracked in #1537. The earlier inline examples used the legacy Cosign format / a pre-bundle Policy Controller version and were removed rather than ship policy that silently fails to verify.

Offline and Air-Gapped Verification

Container image verification uses GitHub’s attestation API (gh attestation verify) because images are already fetched from a registry — an inherently online context. Binary and bundle verification uses sigstore-go with a local trusted root instead. Verification is a read operation that may run frequently — in CI pipelines, in clusters verifying deployed bundles, or by audit tools — and must not be coupled to external API availability or rate limits. Cryptographic security is identical in both cases; the Rekor inclusion proof is embedded in every .sigstore.json bundle and verified locally.

Trusted root management

Bundle verification uses a Sigstore trusted root (CA certificates and Rekor public keys) to validate attestation signatures offline.

Three layers of trust resolution (in priority order):

TUF cache (~/.sigstore/root/) — updated by aicr trust update
Embedded TUF root — compiled into the binary, used to bootstrap
TUF update — aicr trust update contacts the Sigstore TUF CDN

Verification itself never contacts the network — it uses the cache or the embedded root. The install script runs aicr trust update automatically after installation.

$ aicr trust update

Run this when Sigstore rotates their keys (a few times per year) or if verification reports a stale root.