Supply Chain Verification

This guide is for integrators wiring AICR artifact verification into CI pipelines, clusters, and audit tooling. It collects the full command walkthroughs for verifying build provenance, SBOMs, and image/bundle attestations, plus admission-policy enforcement and offline/air-gapped verification.

For a quick trust overview and how to report a vulnerability, see the top-level SECURITY.md.

Prerequisites and Setup

Verification uses Cosign, the GitHub CLI (gh), crane (recommended; docker inspect resolves a digest only after a local pull), jq, ORAS (only for retrieving the third-party source archive), and — for in-cluster enforcement — kubectl. Binary and bundle verification (aicr verify) need only the aicr binary.

Cosign version. AICR release signatures (the binary attestation and the signed recipe catalog) are recorded in Rekor v2 as of the v2 cutover (see the release notes for the exact version). Verifying those bundles with Cosign requires Cosign v3.0.1+: older Cosign cannot parse a Rekor v2 inclusion proof or its RFC3161 timestamp. aicr verify needs only the aicr binary and verifies both v1 and v2 transparently, so the Cosign floor does not apply to it. Releases published before the cutover are in Rekor v1 and verify with any recent Cosign. The verification commands are identical either way: a bundle self-describes which log it is in, so nothing in your workflow changes beyond the Cosign version. For why AICR signs to Rekor v2 and how the signing path works, see Rekor v2 Signing.

Export the following variables once; the rest of this guide reuses them. Tags are mutable and can be repointed to a different image, so resolve the tag to an immutable @sha256: digest and verify against the digest.

$
# Latest release tag
$
export TAG=$(curl -s https://api.github.com/repos/NVIDIA/aicr/releases/latest | jq -r '.tag_name')
$
export VERSION=${TAG#v}  # strip leading 'v' for release filenames
$
$
# CLI image
$
export IMAGE="ghcr.io/nvidia/aicr"
$
export DIGEST=$(crane digest "${IMAGE}:${TAG}")   # crane required; `docker inspect` only resolves a digest after `docker pull`
$
export IMAGE_DIGEST="${IMAGE}@${DIGEST}"
$
export IMAGE_SBOM="$IMAGE:sha256-$(echo "$DIGEST" | cut -d: -f2).sbom"
$
$
# API server image
$
export IMAGE_API="ghcr.io/nvidia/aicrd"
$
export DIGEST_API=$(crane digest "${IMAGE_API}:${TAG}")
$
export IMAGE_API_DIGEST="${IMAGE_API}@${DIGEST_API}"

Authentication (if the registry requires it):

$
docker login ghcr.io

Unified Metadata Retrieval

Release metadata AICR publishes for a container image is attached to that image as an OCI referrer. The first three kinds below are signed in-toto attestations. The source archive is a plain OCI artifact rather than an attestation, but it is Sigstore-signed like the rest, so it is verified with cosign verify instead of cosign verify-attestation. There is one retrieval command per kind, and each has a fixed subject:

The subjects differ because the claims differ. Provenance describes the build that produced the whole release image, and the VEX document is a single, platform-independent triage record keyed by product PURL, so both belong on the index. An SBOM describes exactly one root filesystem, so each platform’s SBOM is attached to that platform’s own manifest digest; querying the index digest for an SBOM returns nothing.

Resolve the digests first. crane digest without --platform returns the index digest; with --platform it returns that platform’s child manifest digest. Resolve the platform digests from the pinned index, not from the tag: a tag repointed between the two lookups would yield a platform digest belonging to a different index than the one the provenance and VEX are verified against, and a wildcard signer pattern would still accept that other release’s validly signed SBOM. This is the same resolution the attest-image-from-tag action performs at publication time.

$
export IMAGE="ghcr.io/nvidia/aicr"
$
export DIGEST=$(crane digest "${IMAGE}:${TAG}")
$
export DIGEST_AMD64=$(crane digest --platform linux/amd64 "${IMAGE}@${DIGEST}")
$
export DIGEST_ARM64=$(crane digest --platform linux/arm64 "${IMAGE}@${DIGEST}")
$
$
export AICR_ISSUER="https://token.actions.githubusercontent.com"
$
# Exact identity, qualified by the release tag. A `refs/tags/.+` regexp would
$
# accept any AICR release's signature, which proves only that some NVIDIA/aicr
>
# release workflow signed the artifact, not that ${TAG} did.
>
export AICR_SIGNER="https://github.com/NVIDIA/aicr/.github/workflows/attest-images.yaml@refs/tags/${TAG}"

Then one command per kind:

$
# Extract under pipefail, and only write a predicate file once the whole
$
# pipeline has succeeded. Both halves are needed: `jq` exits 0 on empty input,
$
# so without pipefail a failed `cosign verify-attestation` still ends the
$
# pipeline with status 0, and the shell truncates a `>` target before cosign
$
# ever runs, so a plain redirect would leave a zero-length file that reads as a
$
# successful extraction.
$
set -o pipefail
$
$
# 1. Build provenance (index digest)
$
gh attestation verify "oci://${IMAGE}@${DIGEST}" \
>
  --repo NVIDIA/aicr \
>
  --signer-workflow NVIDIA/aicr/.github/workflows/attest-images.yaml \
>
  --source-ref "refs/tags/${TAG}" \
>
  --bundle-from-oci
$
$
# 2. SPDX SBOM for one platform (per-platform manifest digest)
$
sbom_amd64=$(cosign verify-attestation --type spdxjson \
>
  --certificate-oidc-issuer "${AICR_ISSUER}" \
>
  --certificate-identity "${AICR_SIGNER}" \
>
  "${IMAGE}@${DIGEST_AMD64}" \
>
  | jq -r '.payload' | base64 -d | jq '.predicate') \
>
  && printf '%s\n' "${sbom_amd64}" > sbom-linux-amd64.spdx.json
$
$
# 3. OpenVEX vulnerability triage (index digest)
$
openvex=$(cosign verify-attestation --type openvex \
>
  --certificate-oidc-issuer "${AICR_ISSUER}" \
>
  --certificate-identity "${AICR_SIGNER}" \
>
  "${IMAGE}@${DIGEST}" \
>
  | jq -r '.payload' | base64 -d | jq '.predicate') \
>
  && printf '%s\n' "${openvex}" > aicr-openvex.json

Each of the seven released images (aicr, aicrd, aicr-gate, and the four aicr-validators/* images) carries the same three kinds; substitute the image name in IMAGE and re-resolve the digests.

Why --bundle-from-oci. gh attestation verify fetches bundles from GitHub’s attestations API unless told otherwise, so the default form would succeed even if the registry referrer push had silently failed. --bundle-from-oci makes it read the same referrer Cosign reads in commands 2 and 3, which is what makes this a single registry-backed retrieval path. Verifying through the API instead is a legitimate fallback when the registry is unreachable or unauthenticated; the sections below that omit the flag are that API-based alternative, and they are labeled as such where they appear.

Cosign version. Commands 2 and 3 are Sigstore bundles published through the OCI referrers path, which requires Cosign v3.0.1+; the same floor the Rekor v2 note above sets. Command 1 uses the GitHub CLI and does not involve Cosign, so the floor does not apply to it. GHCR does not implement the OCI 1.1 /v2/{name}/referrers/{digest} endpoint, so clients fall back to the specification’s referrers tag schema (a sha256-{hex} tag holding an index of referrers). Cosign and ORAS do this transparently, which is why the commands above are the documented path rather than a raw referrers API call.

Third-Party Source Code

Source for every third-party open-source component AICR adds on top of its base images is published, regardless of license. Where it lives depends on how the component reaches the image.

License texts for all of the above are in THIRD_PARTY_NOTICES.md, published as an asset on each release. That file discharges attribution; this section covers source availability, which is a separate obligation.

Retrieving the aiperf-bench source

The archive is attached as an OCI referrer, so it is not fetched by docker pull and will not appear in the image’s layers. Retrieve it explicitly:

$
IMAGE="ghcr.io/nvidia/aicr-validators/aiperf-bench:v0.19.0"
$
$
# 1. Resolve the image to its digest and list what is attached to it.
$
DIGEST=$(crane digest "${IMAGE}")
$
oras discover --format tree "${IMAGE%:*}@${DIGEST}"
$
$
# 2. Select the source referrer. Fail loudly on anything but exactly one:
$
#    artifactType is not a unique key, and anyone with push access to the
$
#    repository can attach another referrer carrying the same type. Taking the
$
#    first match would silently pick theirs.
$
mapfile -t MATCHES < <(oras discover --format json "${IMAGE%:*}@${DIGEST}" \
>
  | jq -r '.referrers[]
>
           | select(.artifactType == "application/vnd.nvidia.aicr.source.v1+tar")
>
           | .digest')
$
if [ "${#MATCHES[@]}" -ne 1 ]; then
$
  echo "expected exactly 1 source referrer, found ${#MATCHES[@]}" >&2
$
  exit 1
$
fi
$
SOURCE="${MATCHES[0]}"
$
$
# 3. Verify the signature before trusting the bytes. Pin the exact signing
$
#    workflow and tag; a regexp over the repository would accept a signature
$
#    from any workflow in it.
$
cosign verify \
>
  --certificate-oidc-issuer "${AICR_ISSUER}" \
>
  --certificate-identity \
>
    "https://github.com/NVIDIA/aicr/.github/workflows/on-tag.yaml@refs/tags/${TAG}" \
>
  "${IMAGE%:*}@${SOURCE}"
$
$
# 4. Pull the verified archive.
$
oras pull -o ./aiperf-source "${IMAGE%:*}@${SOURCE}"
$
$
# 5. The archive holds one sdist per installed package, plus a README.
$
tar tzf ./aiperf-source/aiperf-bench-python-source.tar.gz | head

The archive contains the source distribution of every Python package installed into the image that publishes one upstream, at the exact version installed. One package is intentionally absent: aiperf itself publishes no source distribution to PyPI. It is an NVIDIA package and its source is at ai-dynamo/aiperf; the archive’s README.txt records this too.

Only aiperf-bench carries this referrer. The other six images need none: their dependencies are vendored, so the release source archive already contains them.

Scope of the correspondence. The archive is resolved from the same requirements.txt the image installs, so both derive from one input. Two gaps remain and are tracked rather than claimed away:

• pip and wheel are provided by python -m venv rather than declared in requirements.txt, so they ship in the image without source in this archive.
• The image and the archive resolve the ranged dependencies independently: the image when build-docker runs pip install, the archive later in attach-source. A transitive release landing between the two resolves the archive newer than the image, so the closures can differ by that window.

Correspondence is therefore by shared input, not by identical resolved closure. An exact guarantee would require resolving the archive from the built image’s own pip freeze, or pinning both to a shared lockfile.

The second gap used to be unbounded. The image build layer-cached its resolution, so a cached build could replay a months-old closure (#2086); it now resolves fresh on every build, narrowing the divergence to the build-to-attach window.

A referrer binds to one image digest, so resolve the tag to a digest first as shown above rather than assuming a tag keeps the same attachment across builds.

Using the published VEX document

The OpenVEX document records, per CVE, why AICR is not affected: a machine readable status and justification plus a human-readable impact statement with the reachability evidence behind the call. Feeding it to your scanner suppresses exactly the findings AICR’s own release scan suppresses, so a downstream gate does not re-flag CVEs that have already been triaged.

$
# Grype: apply the VEX document to a scan of the same image
$
grype "${IMAGE}@${DIGEST}" --vex aicr-openvex.json --only-fixed --fail-on high
$
$
# Trivy: same document, same effect
$
trivy image --vex aicr-openvex.json "${IMAGE}@${DIGEST}"

Statements apply only to products whose PURL matches, so passing the document to a scan of an unrelated image is a no-op rather than a blanket suppression. Treat the document as evidence, not as an instruction: read the impact_statement for any CVE your policy cares about and decide whether AICR’s reasoning holds for your deployment before adopting the suppression.

Why provenance uses a different signer

Image provenance is produced by actions/attest-build-provenance from the reusable attest-images.yaml workflow, while the SBOM and VEX attestations are produced by cosign attest in the same job. Unifying all three onto cosign attest would make the signer identity uniform but would drop the image provenance from SLSA Build Level 3 to Level 2: the Level 3 isolation property comes from GitHub’s attestation service recording the reusable workflow as the builder identity, which a caller workflow cannot forge. AICR keeps the GitHub attestation flow for provenance for that reason. All three land on the same image as OCI referrers regardless, so the split affects which verifier you reach for, not where the metadata lives.

Verifying Build Provenance (SLSA)

AICR produces SLSA build provenance through GitHub Actions: builds are defined as code and provenance is service-generated (signed by GitHub’s OIDC-authenticated attestation service via actions/attest-build-provenance) rather than self-asserted, then logged to the public Rekor transparency log.

Note on the SLSA Build Level. GitHub’s attestation service yields Build Level 2 by default; Build Level 3 additionally requires build isolation via a dedicated reusable workflow. AICR generates image attestations from the reusable attest-images.yaml workflow, so the image provenance is Build Level 3 — its signer identity is that reusable workflow, which the caller cannot tamper with. Verify it by pinning --signer-workflow .../attest-images.yaml (below). CLI binaries are signed with cosign attest-blob from the release job and remain Build Level 2.

Method 1: GitHub CLI

These commands omit --bundle-from-oci, so they are the API-based alternative: gh fetches the bundle from GitHub’s attestations API rather than from the registry referrer. Add --bundle-from-oci to verify the referrer itself, as Unified Metadata Retrieval does.

$
# Verify provenance exists and is valid (using digest)
$
gh attestation verify oci://${IMAGE_DIGEST} --repo NVIDIA/aicr --signer-workflow NVIDIA/aicr/.github/workflows/attest-images.yaml --source-ref "refs/tags/${TAG}"
$
$
# Output shows:
$
# ✓ Verification succeeded!
$
#
$
# Attestations:
$
#   • Build provenance (SLSA v1.0)
$
# (the SPDX SBOM is a separate Cosign attestation — see Verifying the SBOM)

Method 2: Extract and inspect provenance

$
# Get full provenance data (using digest)
$
gh attestation verify oci://${IMAGE_DIGEST} \
>
  --repo NVIDIA/aicr --signer-workflow NVIDIA/aicr/.github/workflows/attest-images.yaml --source-ref "refs/tags/${TAG}" \
>
  --format json | jq '.[] | select(.verificationResult.statement.predicateType | contains("slsa"))'
$
$
# Key fields in provenance:
$
# - buildDefinition.buildType: GitHub Actions workflow type
$
# - runDetails.builder.id: Workflow file and commit
$
# - buildDefinition.externalParameters.workflow: Workflow path and ref
$
# - buildDefinition.resolvedDependencies: Source code commit SHA
$
# - runDetails.metadata.invocationId: GitHub run ID

The signed certificate binds the artifact to its source repository, commit SHA, workflow, and run. A representative slice:

1
{
2
  "verificationResult": {
3
    "signature": {
4
      "certificate": {
5
        "subjectAlternativeName": "https://github.com/NVIDIA/aicr/.github/workflows/attest-images.yaml@refs/tags/v0.8.12",
6
        "issuer": "https://token.actions.githubusercontent.com",
7
        "githubWorkflowName": "on_tag",
8
        "githubWorkflowRepository": "NVIDIA/aicr",
9
        "githubWorkflowRef": "refs/tags/v0.8.12",
10
        "sourceRepositoryURI": "https://github.com/NVIDIA/aicr",
11
        "sourceRepositoryDigest": "ba6cbbe8b1a8fc8b72bb18454c10a3ba31d94a2e",
12
        "runnerEnvironment": "github-hosted",
13
        "runInvocationURI": "https://github.com/NVIDIA/aicr/actions/runs/20642050863/attempts/1"
14
      }
15
    }
16
  }
17
}

Build process transparency

All AICR releases are built using GitHub Actions with full transparency:

1. Source Code — Public GitHub repository
2. Build & Attest Workflows — .github/workflows/on-tag.yaml builds and calls the reusable .github/workflows/attest-images.yaml, which signs the image attestations (both version controlled)
3. Build Logs — Public GitHub Actions run logs
4. Attestations — Signed and stored in the public transparency log (Rekor)
5. Artifacts — Published to GitHub Releases and GHCR

View build history:

$
# List all releases with attestations
$
gh api repos/NVIDIA/aicr/releases | \
>
  jq -r '.[] | "\(.tag_name): \(.html_url)"'
$
$
# View specific build logs
$
gh run list --repo NVIDIA/aicr --workflow=on-tag.yaml
$
gh run view 20642050863 --repo NVIDIA/aicr --log

Verify in the transparency log (Rekor):

$
# Search Rekor for attestations
$
rekor-cli search --sha "${DIGEST#sha256:}"
$
$
# Get entry details
$
rekor-cli get --uuid <entry-uuid>

Verifying the SBOM

AICR provides SBOMs in SPDX v2.3 JSON format: binary SBOMs as separate GoReleaser artifacts (generated alongside CLI binaries) and container image SBOMs attached as Cosign attestations (generated by Syft/Anchore).

Binary SBOM (CLI)

$
# Detect OS and architecture
$
export OS=$(uname -s | tr '[:upper:]' '[:lower:]')
$
export ARCH=$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/')
$
$
# Download the versioned archive from GitHub releases and extract the binary.
$
# GoReleaser ships aicr_<version>_<os>_<arch>.tar.gz; ${VERSION} is the tag
$
# without its leading "v" (e.g. TAG=v0.8.12 -> VERSION=0.8.12).
$
curl -LO https://github.com/NVIDIA/aicr/releases/download/${TAG}/aicr_${VERSION}_${OS}_${ARCH}.tar.gz
$
tar -xzf aicr_${VERSION}_${OS}_${ARCH}.tar.gz
$
chmod +x aicr
$
$
# Download SBOM (separate file)
$
curl -LO https://github.com/NVIDIA/aicr/releases/download/${TAG}/aicr_${VERSION}_${OS}_${ARCH}.sbom.json
$
$
# View SBOM
$
cat aicr_${VERSION}_${OS}_${ARCH}.sbom.json

Each binary SBOM ships with its own Sigstore bundle, aicr_${VERSION}_${OS}_${ARCH}.sbom.json.sigstore.json, carrying the release run’s SLSA provenance over the SBOM document itself. Download it alongside the SBOM and verify that NVIDIA CI produced that exact document:

$
curl -LO https://github.com/NVIDIA/aicr/releases/download/${TAG}/aicr_${VERSION}_${OS}_${ARCH}.sbom.json.sigstore.json
$
$
cosign verify-blob-attestation \
>
  --bundle aicr_${VERSION}_${OS}_${ARCH}.sbom.json.sigstore.json \
>
  --type https://slsa.dev/provenance/v1 \
>
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>
  --certificate-identity-regexp '^https://github\.com/NVIDIA/aicr/\.github/workflows/on-tag\.yaml@refs/tags/.+$' \
>
  aicr_${VERSION}_${OS}_${ARCH}.sbom.json

Binary metadata stays on the GitHub release rather than getting its own OCI home: the binaries are distributed as release assets, so a Sigstore bundle downloaded next to the asset it covers keeps the artifact and its evidence on a single retrieval path. Image metadata lives in the registry for the same reason.

Container image SBOM

Container image SBOMs are attached per platform: each platform’s SPDX document is an attestation on that platform’s manifest digest, not on the multi-platform index digest. Resolve the platform digest with crane digest --platform (see Unified Metadata Retrieval) before verifying.

$
# pipefail plus the deferred write, for the reason given under
$
# Unified Metadata Retrieval: a failed verification must not leave an empty
$
# sbom.json behind.
$
set -o pipefail
$
$
# Resolve from the pinned index (${DIGEST_API}), never from the mutable tag.
$
export DIGEST_API_AMD64=$(crane digest --platform linux/amd64 "${IMAGE_API}@${DIGEST_API}")
$
$
# Method 1: Using Cosign (extracts attestation) - uses the per-platform digest
$
sbom=$(cosign verify-attestation \
>
  --type spdxjson \
>
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>
  --certificate-identity-regexp '^https://github\.com/NVIDIA/aicr/\.github/workflows/attest-images\.yaml@refs/tags/.+$' \
>
  ${IMAGE_API}@${DIGEST_API_AMD64} | \
>
  jq -r '.payload' | base64 -d | jq '.predicate') \
>
  && printf '%s\n' "${sbom}" > sbom.json
$
$
# Method 2: GitHub CLI (build provenance only; the SPDX SBOM needs Method 1's Cosign flow).
>
# Without --bundle-from-oci this reads GitHub's attestations API, not the registry referrer.
>
gh attestation verify oci://${IMAGE_API_DIGEST} --repo NVIDIA/aicr --signer-workflow NVIDIA/aicr/.github/workflows/attest-images.yaml --source-ref "refs/tags/${TAG}" --format json

SBOM format

Both binary and container SBOMs are SPDX v2.3 JSON. A representative package entry (the full document lists every Go module and its transitive dependencies, licenses, and package URLs):

1
{
2
  "spdxVersion": "SPDX-2.3",
3
  "dataLicense": "CC0-1.0",
4
  "name": "aicr",
5
  "creationInfo": {
6
    "creators": ["Organization: Anchore, Inc", "Tool: syft-1.38.2"]
7
  },
8
  "packages": [
9
    {
10
      "name": "github.com/NVIDIA/aicr",
11
      "versionInfo": "v0.8.12",
12
      "externalRefs": [
13
        {
14
          "referenceType": "purl",
15
          "referenceLocator": "pkg:golang/github.com/NVIDIA/aicr@v0.8.12"
16
        }
17
      ]
18
    }
19
  ]
20
}

SBOM use cases

$
# Vulnerability scanning — feed the SBOM to Grype, Anchore, or Snyk
$
grype sbom:./sbom.json
$
$
# License compliance — list declared licenses
$
jq -r '.packages[] | select(.licenseDeclared != "NOASSERTION") | "\(.name) \(.versionInfo) \(.licenseDeclared)"' sbom.json
$
$
# Dependency tracking — search for a specific component
$
jq '.packages[] | select(.name | contains("vulnerable-lib"))' sbom.json
$
$
# Audit trail — the SBOM timestamp proves when components were included
$
jq '.creationInfo.created' sbom.json

Verifying Image and Bundle Attestations

Container image attestations

Method 1: GitHub CLI (recommended)

Also the API-based path: append --bundle-from-oci to each command to verify the registry referrer instead of GitHub’s attestations API.

$
# Verify using digest (preferred - no warnings)
$
gh attestation verify oci://${IMAGE_DIGEST} --repo NVIDIA/aicr --signer-workflow NVIDIA/aicr/.github/workflows/attest-images.yaml --source-ref "refs/tags/${TAG}"
$
$
# Verify the aicrd image
$
gh attestation verify oci://${IMAGE_API_DIGEST} --repo NVIDIA/aicr --signer-workflow NVIDIA/aicr/.github/workflows/attest-images.yaml --source-ref "refs/tags/${TAG}"
$
$
# Note: You can still use tags, but tools may show warnings about mutability
$
# gh attestation verify oci://ghcr.io/nvidia/aicr:${TAG} --repo NVIDIA/aicr --signer-workflow NVIDIA/aicr/.github/workflows/attest-images.yaml --source-ref "refs/tags/${TAG}"

Method 2: Cosign (SBOM and VEX attestations)

$
# pipefail plus the deferred write, for the reason given under
$
# Unified Metadata Retrieval: a failed verification must not leave an empty
$
# predicate file behind.
$
set -o pipefail
$
$
# Verify the SBOM attestation on a per-platform manifest digest,
$
# resolved from the pinned index rather than from the mutable tag.
$
export DIGEST_AMD64=$(crane digest --platform linux/amd64 "${IMAGE}@${DIGEST}")
$
cosign verify-attestation \
>
  --type spdxjson \
>
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>
  --certificate-identity-regexp '^https://github\.com/NVIDIA/aicr/\.github/workflows/attest-images\.yaml@refs/tags/.+$' \
>
  ${IMAGE}@${DIGEST_AMD64}
$
$
# Extract and view the SBOM predicate
$
cosign verify-attestation \
>
  --type spdxjson \
>
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>
  --certificate-identity-regexp '^https://github\.com/NVIDIA/aicr/\.github/workflows/attest-images\.yaml@refs/tags/.+$' \
>
  ${IMAGE}@${DIGEST_AMD64} | jq -r '.payload' | base64 -d | jq '.predicate'
$
$
# Verify the OpenVEX attestation on the index digest
$
openvex=$(cosign verify-attestation \
>
  --type openvex \
>
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>
  --certificate-identity-regexp '^https://github\.com/NVIDIA/aicr/\.github/workflows/attest-images\.yaml@refs/tags/.+$' \
>
  ${IMAGE_DIGEST} | jq -r '.payload' | base64 -d | jq '.predicate') \
>
  && printf '%s\n' "${openvex}" > aicr-openvex.json

CLI binary attestation

CLI binary releases are attested with SLSA Build Provenance v1 using Cosign keyless signing via GitHub Actions OIDC. Each release archive (.tar.gz) contains the aicr binary and an aicr-attestation.sigstore.json Sigstore bundle. The attestation is logged to the public Rekor transparency log and can be verified offline.

$
cosign verify-blob-attestation \
>
  --bundle aicr-attestation.sigstore.json \
>
  --type https://slsa.dev/provenance/v1 \
>
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>
  --certificate-identity-regexp '^https://github\.com/NVIDIA/aicr/\.github/workflows/on-tag\.yaml@refs/tags/.+$' \
>
  aicr

The install script (./install) runs this verification automatically when Cosign is available. The Build Attested Binaries workflow (.github/workflows/build-attested.yaml) can be triggered manually from the Actions tab to produce attested binaries from any branch without cutting a release.

Bundle attestation

When aicr bundle runs with --attest, it signs the bundle using Sigstore keyless OIDC, binding the bundle creator’s identity to the generated closed-world inventory and the binary that produced it (via resolvedDependencies). Attestation is opt-in; bundles are unsigned by default. The bundle output includes attestation/bundle-attestation.sigstore.json (SLSA Build Provenance v1 for the bundle) and attestation/aicr-attestation.sigstore.json (the binary provenance chain).

Bundle verification is closed-world: checksums.txt defines the regular payload, and any unlisted or non-regular entry fails verification. See Artifact Verification for the exact metadata exceptions, ordering rules, publication guarantees, and legacy-bundle behavior.

$
aicr verify ./my-bundle

This performs full closed-world verification: it validates every manifest digest and rejects any additional filesystem entry, then verifies the bundle attestation against the Sigstore trusted root and the binary attestation provenance chain (identity pinned to NVIDIA CI). Manifest parsing is order-independent, but reordering an already signed checksums.txt changes the signed bytes and invalidates its existing attestation. Enforce a minimum trust level:

$
aicr verify ./my-bundle --min-trust-level verified

For full CLI flag documentation, see the CLI Reference (aicr verify, aicr bundle --attest, aicr trust update). For a hands-on walkthrough, see the Bundle Attestation Demo.

Enforcing with Admission Policies

You can enforce provenance verification at deployment time with a Kubernetes admission controller. AICR’s images carry GitHub Artifact Attestations, which are Sigstore bundles — so the admission policy must verify the Sigstore bundle format (not the legacy Cosign signature format):

Pin every policy to AICR’s release identity:

• issuer: https://token.actions.githubusercontent.com
• subject: https://github.com/NVIDIA/aicr/.github/workflows/attest-images.yaml@refs/tags/* (the reusable attestation workflow that signs image provenance/SBOMs; narrow to the release pattern rather than trusting every workflow/ref)

Kyverno

Not verified against AICR images — use Sigstore Policy Controller (below). Kyverno verifies Sigstore bundles with type: SigstoreBundle (v1.18+; see Kyverno’s Verifying Sigstore Bundles guide). In testing on GKE 1.35 with Kyverno v1.18.1, a SigstoreBundle verifyImages rule pinned to AICR’s release identity could not verify AICR’s GitHub Artifact Attestation — it failed with no matching signatures found, even though cosign verify-attestation and the Policy Controller policy below verify the same Sigstore-bundle (v0.3) referrer on the image’s index digest. Until that gap is understood, enforce AICR images with the Sigstore Policy Controller policy below. Tracking: #1537.

Sigstore Policy Controller

Sigstore-bundle support requires v0.13.0+ and signatureFormat: bundle; see the Sigstore bundle format docs. Enforcement only runs in namespaces labeled policy.sigstore.dev/include=true.

1
apiVersion: policy.sigstore.dev/v1beta1
2
kind: ClusterImagePolicy
3
metadata:
4
  name: aicr-require-provenance
5
spec:
6
  images:
7
    - glob: "ghcr.io/nvidia/aicr**"
8
  authorities:
9
    - name: aicr-release
10
      signatureFormat: bundle
11
      keyless:
12
        url: https://fulcio.sigstore.dev
13
        identities:
14
          - issuer: https://token.actions.githubusercontent.com
15
            subjectRegExp: '^https://github\.com/NVIDIA/aicr/\.github/workflows/attest-images\.yaml@refs/tags/.+$'
16
      ctlog:
17
        url: https://rekor.sigstore.dev
18
      attestations:
19
        - name: slsa-provenance
20
          predicateType: https://slsa.dev/provenance/v1

Save the ClusterImagePolicy above as clusterimagepolicy.yaml, apply it, create and label a target namespace, then confirm enforcement (DIGEST is resolved in Prerequisites and Setup above):

$
kubectl apply -f clusterimagepolicy.yaml
$
kubectl -n cosign-system rollout status deploy/policy-controller-webhook
$
export NAMESPACE=aicr-policy-test
$
kubectl create namespace "$NAMESPACE"
$
kubectl label namespace "$NAMESPACE" policy.sigstore.dev/include=true
$
sleep 15   # let the webhook ingest the new policy
$
$
# Positive: a signed AICR image (pinned by digest) is admitted
$
kubectl -n "$NAMESPACE" run aicr-signed \
>
  --image="ghcr.io/nvidia/aicr@${DIGEST}" --restart=Never \
>
  --command -- /ko-app/aicr --version

For a coherent negative test the image must match the policy glob (ghcr.io/nvidia/aicr**) yet be unsigned — a non-matching image is simply ignored by the policy. Push an unsigned image under a path you control whose name the glob matches (or temporarily widen the glob to it), then confirm the admission webhook rejects it.

Validation status. The Policy Controller ClusterImagePolicy above is cluster-validated against Policy Controller v0.13.1 on GKE 1.35: a signed AICR image is admitted and a wrong-identity pin is rejected (note that signatureFormat and ctlog are per-authority fields). The Kyverno SigstoreBundle path was cluster-tested (v1.18.1) and failed to verify AICR’s bundle attestation (no matching signatures found) — see the Kyverno note above; tracked in #1537.

Gating Deployment on Verification

aicr verify is the deploy-time gate: run it against the bundle directory before anything installs that bundle or publishes it for a controller to pull. Because checksums.txt is a closed-world inventory of every payload file, verifying it transitively covers the whole bundle; see Artifact Verification.

Where that gate belongs depends on whether the deployer pushes or pulls, so each one is covered separately below. Every gate is marked either advisory (a pipeline step that an operator holding cluster credentials can bypass) or binding (the cluster itself refuses to proceed).

Gating the helm and helmfile deployers

The default helm deployer and helmfile are push-based: the same pipeline step that verifies is the one that installs, so verification and use cannot drift apart.

$
cd bundles
$
aicr verify . --min-trust-level verified
$
chmod +x deploy.sh
$
./deploy.sh

aicr verify exits non-zero on any failure, so && chaining or set -e is enough to stop the install. Prefer an explicit --min-trust-level verified over the default max, which resolves to the highest level that particular bundle can reach and therefore passes an unsigned bundle. Add --require-creator to pin who built it and --cli-version-constraint to floor the aicr version that produced it; --format json emits a machine-readable result for a pipeline that needs to branch. For helmfile, run the same aicr verify . before helmfile apply. See Automation for the full four-stage pipeline and its GitLab, CircleCI, and Terraform equivalents.

These gates are advisory. They run in your pipeline, on the machine that holds the kubeconfig. Anyone who can run helm install directly, or who can edit the pipeline definition, bypasses them. Protect them the way you protect any pipeline: branch protection on the workflow definition, and cluster credentials that only the pipeline holds.

Gating Argo CD

Argo CD is pull-based. aicr bundle --deployer argocd --repo <git-url> writes app-of-apps.yaml plus per-component NNN-<component>/application.yaml, you commit them, and the cluster’s Argo CD reconciles the repository on its own schedule. A CI step therefore gates what enters the repository, not what the cluster applies. Verify before you commit:

$
aicr bundle --recipe recipe.yaml --deployer argocd --attest \
>
  --repo https://github.com/my-org/my-gitops-repo.git \
>
  --output ./bundles
$
aicr verify ./bundles --min-trust-level verified
$
cp -r ./bundles/* gitops-repo/
$
(cd gitops-repo && git add . && git commit -S -m "Update GPU stack" && git push)

Verify the bundle directory, never the GitOps repository. Verification is closed-world and rejects any filesystem entry not listed in checksums.txt, so running it against a repository that holds anything else fails.

For a binding control, have the pipeline sign the commit and have Argo CD refuse unsigned revisions. AppProject.spec.signatureKeys lists the GnuPG key IDs allowed to sign; Argo CD then refuses to sync any revision that is unsigned or signed by a key outside that list:

1
apiVersion: argoproj.io/v1alpha1
2
kind: AppProject
3
metadata:
4
  name: gpu-stack
5
  namespace: argocd
6
spec:
7
  signatureKeys:
8
    - keyID: 4AEE18F83AFDEB23
9
  sourceRepos:
10
    - https://github.com/my-org/my-gitops-repo.git

Read the trust chain carefully. The cluster enforces “this revision was signed by CI,” and CI signs only after aicr verify passed. The cluster is not verifying the AICR attestation; it is trusting your pipeline’s identity to have checked it. Argo CD’s GnuPG verification applies to Git sources only, not to Helm repositories, and it disables argocd app sync --local. See Argo CD’s GnuPG signature verification.

Gating Flux

Flux is likewise pull-based, in two source shapes.

Git source (default). aicr bundle --deployer flux --output ./flux-bundle writes a root kustomization.yaml, a sources/ directory, per-component helmrelease.yaml files, and a README.md carrying the entry-point GitRepository and Kustomization you apply to the cluster. Verify the bundle directory before copying it to the repository root, exactly as for Argo CD. Bind the reconcile by signing the commit in CI and adding spec.verify to that entry-point GitRepository:

1
apiVersion: source.toolkit.fluxcd.io/v1
2
kind: GitRepository
3
metadata:
4
  name: aicr-stack
5
  namespace: flux-system
6
spec:
7
  url: https://github.com/my-org/my-gitops-repo.git
8
  ref:
9
    branch: main
10
  interval: 10m
11
  verify:
12
    mode: HEAD
13
    secretRef:
14
      name: pgp-public-keys

mode: HEAD verifies the commit at the checked-out HEAD; Tag and TagAndHEAD are the other accepted values. The referenced Secret holds the trusted public keys (.asc for PGP, .sshpub for SSH). See Flux’s GitRepository verification.

OCI source. With --output oci://..., AICR pushes the bundle as an OCI artifact and generates ArtifactGenerator CRs that reference an OCIRepository you deploy (named by --flux-oci-source-name, default aicr-bundle, in --flux-namespace, default flux-system). Because you own that OCIRepository, you can require a Cosign signature on the artifact before Flux will reconcile anything from it:

1
apiVersion: source.toolkit.fluxcd.io/v1
2
kind: OCIRepository
3
metadata:
4
  name: aicr-bundle
5
  namespace: flux-system
6
spec:
7
  interval: 10m
8
  url: oci://ghcr.io/my-org/aicr-bundle
9
  verify:
10
    provider: cosign
11
    matchOIDCIdentity:
12
      - issuer: "^https://token\\.actions\\.githubusercontent\\.com$"
13
        subject: "^https://github\\.com/my-org/my-gitops-repo/\\.github/workflows/deploy\\.yaml@refs/heads/main$"

That Cosign signature is not AICR’s bundle attestation, and AICR does not produce it: your pipeline countersigns the pushed manifest after aicr verify passes. An OCI push also materializes the same inventory in ./bundle relative to the working directory, which is the copy to verify, and --image-refs captures the published digest so the countersignature pins the exact manifest:

$
aicr bundle --recipe recipe.yaml --deployer flux --attest \
>
  --output oci://ghcr.io/my-org/aicr-bundle:v1.0.0 \
>
  --image-refs ./published-digest.txt
$
aicr verify ./bundle --min-trust-level verified
$
cosign sign "ghcr.io/my-org/aicr-bundle@$(cat ./published-digest.txt)"

This one is binding: source-controller will not produce an artifact from an OCIRepository whose signature fails to verify, so no HelmRelease downstream of it reconciles. As with Argo CD, what the cluster enforces is your pipeline’s countersignature, not the AICR attestation. Flux OCI mode also has its own prerequisites (Flux v2.7+ with source-watcher and the ExternalArtifact feature gate); see Flux OCI Mode and Flux’s OCIRepository verification.

Gates across trust environments

Only the trust material handed to aicr verify changes between environments; the gate itself is the same command in the same place.

--trust-root is additive to AICR’s built-in public-good root, so one command verifies both org-signed and NVIDIA-signed bundles. --insecure-ignore-tlog requires --key, and a local PEM key keeps the verify fully offline where a KMS URI still resolves remotely. Full details for each shape are in Artifact Verification.

The binding half is unchanged across all four rows, because it enforces your pipeline’s own signature rather than AICR’s. Air-gapped sites should note that Flux’s spec.verify keyless form depends on public Sigstore: use a key-based Cosign signature with spec.verify.secretRef instead of matchOIDCIdentity.

Offline and Air-Gapped Verification

Container image verification uses GitHub’s attestation API (gh attestation verify) because images are already fetched from a registry — an inherently online context. Binary and bundle verification uses sigstore-go with a local trusted root instead. Verification is a read operation that may run frequently — in CI pipelines, in clusters verifying deployed bundles, or by audit tools — and must not be coupled to external API availability or rate limits. Cryptographic security is identical in both cases; the Rekor inclusion proof is embedded in every .sigstore.json bundle and verified locally.

Trusted root management

Bundle verification uses a Sigstore trusted root (CA certificates and Rekor public keys) to validate attestation signatures offline.

Three layers of trust resolution (in priority order):

1. TUF cache (~/.sigstore/root/) — updated by aicr trust update
2. Embedded TUF root — compiled into the binary, used to bootstrap
3. TUF update — aicr trust update contacts the Sigstore TUF CDN

Verification itself never contacts the network — it uses the cache or the embedded root. The install script runs aicr trust update automatically after installation.

$
aicr trust update

Run this when Sigstore rotates their keys (a few times per year) or if verification reports a stale root.

Monitoring Your Signing Identity

Everything above is consumer-side: it proves that an artifact you already hold came from the identity you expect. It cannot tell you that somebody else signed something as you. That is the producer-side question, and the transparency log is the only place it can be answered. An entry under your signing identity that you did not produce may indicate that the identity was used without you, and it is the only signal that will tell you.

The gap is sharpest for keyless signing, which leaves no local trace. The Fulcio certificate is short-lived, there is no private key on disk whose use you could audit, and the signing event is invisible once the process exits. The Rekor entry is the only durable record, so watching the log is the only way to notice misuse. Sigstore reaches the same conclusion: its threat model treats identity and consistency monitoring as the detection control for a compromised signing identity, and its Rekor documentation tells artifact owners to monitor the log for their own identity. AICR runs exactly this check against its own release signing identity in .github/workflows/rekor-monitor.yaml; this section is how to run it against yours.

Every AICR signing path uploads to a transparency log by default: aicr bundle --attest, aicr validate --emit-attestation --push, aicr evidence publish, and aicr evidence sign. The one signing mode that uploads nothing is --tlog-upload=false (KMS-only air-gapped signing). With no log entry there is nothing to monitor.

What your signature records

Configure the monitor from what your own entries actually carry, rather than guessing. Read it off a bundle you signed:

$
# Fulcio certificate from a keyless-signed bundle. The SAN is under
$
# "X509v3 Subject Alternative Name"; the OIDC issuer is extension
$
# 1.3.6.1.4.1.57264.1.8.
$
jq -r '.verificationMaterial.certificate.rawBytes' \
>
  ./my-bundle/attestation/bundle-attestation.sigstore.json \
>
  | base64 -d | openssl x509 -inform DER -noout -text

The flags in the first column are aicr bundle flags. --signing-key, --fulcio-url, --rekor-url, --signing-config, and --tlog-upload are registered on that command only; the evidence-signing paths (aicr validate --emit-attestation --push, aicr evidence publish, aicr evidence sign) are keyless-only and always write to the default Rekor v2, with no key and no Rekor v1 override.

Note the third column carefully. Rekor v2 is the default for every keyless and KMS signing path in the CLI, and only --rekor-url and --signing-config change the log. --fulcio-url selects the certificate authority, not the log, so a private CA on its own still publishes into the public-good Rekor v2. Pair it with --rekor-url to keep the entry inside your own infrastructure. See Rekor v2 Signing. Several of these rows have no working monitor today; the coverage gaps at the end of this section say which and why.

Monitoring the public-good Rekor v2

The upstream sigstore/rekor-monitor reusable workflow cannot monitor this log today. It resolves which Rekor to read from Sigstore’s default signing config, which lists only Rekor v1; Sigstore’s Rekor evolution post states that the public-good instance will continue using Rekor v1 as the default log for the foreseeable future. The workflow’s inputs are file_issue, artifact_retention_days, once, config, and url; none of them selects a different signing config, and pointing url at a v2 shard falls through to its v1 client and fails.

AICR hit that same wall and wrote tools/rekor-monitor for it. The tool resolves the v2 shard from the same signing config AICR signs against, then delegates the security-critical verification to upstream’s library packages. It takes the watched identity as flags, so it monitors your identity as readily as AICR’s:

$
# Pin the tool you audit by commit SHA. Cloning the default branch means the
$
# monitor can change under you between runs; bump this deliberately after
$
# review. `git clone --branch` resolves only branch and tag names, so the
$
# checkout is a separate step.
$
git clone https://github.com/NVIDIA/aicr && cd aicr
$
git checkout --detach d4f7bef460dc8d1ef7ea0334a6935c0038de88e4
$
$
# Allowlist the tags a real release actually signed, so a legitimate release
$
# does not alert. Derive it from your release workflow's completed run history,
>
# never from `git tag --list`: an attacker-pushed or never-signed tag present in
>
# the repository would be pre-suppressed. The workflow example below shows the
>
# query; run it once ahead of the monitor.
>
GOFLAGS="-mod=vendor" go run ./tools/rekor-monitor \
>
  --file checkpoint_v2.txt \
>
  --cert-subject '^https://github\.com/myorg/myrepo/\.github/workflows/release\.yaml@refs/tags/.*$' \
>
  --cert-issuer '^https://token\.actions\.githubusercontent\.com$' \
>
  --known-tags-file known-tags.txt

The two halves of the identity must come off the same certificate. Under the GitHub Actions issuer the SAN is the workflow identity URL shown above, never an email address; an email SAN belongs with the interactive IdP issuer (^https://oauth2\.sigstore\.dev/auth$ for the default browser flow). Subject and issuer are AND-ed, so an incoherent pair matches nothing at all and the monitor reports clean forever while watching an identity that cannot exist.

--cert-subject and --cert-issuer are regexes matched against the certificate’s SAN and OIDC issuer extension; anchor them, or a lookalike identity matches. --file is the cursor: the first run baselines at the current tree head and scans nothing, and every later run scans only the window added since. The tool writes two companions alongside it. <file>.scan holds how far the current window has been scanned, so a large backlog is caught up across several bounded runs; it must survive between runs alongside the checkpoint, or the window is rescanned from the start. <file>.stall holds only the catch-up convergence history that feeds the degraded classification, so losing it costs stall detection, not scan progress. --timeout bounds a single pass. Exit 0 is clean, 1 is a security finding (tamper or identity), and 3 is a non-security failure (operational for Sigstore, Rekor, or network trouble; degraded when the log is outpacing the per-run scan). Every completed run prints a matching CLASSIFICATION= line to branch on. Full flag and exit-code reference: tools/rekor-monitor/README.md.

Know the one limitation before you rely on this. On an identity match the tool deliberately holds the cursor before the matching chunk rather than advancing past it, because advancing would let the next clean window auto-close the alert without anyone acknowledging it. The consequence is that the same finding is re-detected and re-alerted on every subsequent run until a maintainer triages it. The only built-in suppression is --known-tags-file, and it works by matching a release tag inside the certificate SAN, so it applies to a tag-bearing release identity and not to an email or a generic CI identity. The tool is therefore turnkey for an identity that signs rarely enough to triage each hit one at a time, and for a tag-bearing release identity via --known-tags-file. For an identity that signs continuously it is not: its first legitimate signature after the baseline leaves the monitor permanently alerting, and closing that out needs an acknowledgment mechanism the tool does not have yet.

Both examples here watch a tag-bearing release identity, so both pass --known-tags-file. Two residual gaps come with that suppression. An attacker who re-signs an existing release tag is suppressed, because that tag is legitimately on the allowlist. And the allowlist keys on a completed release run rather than on proof that the run’s signing step succeeded, since signing happens mid-run and a release that flaked afterwards still produced a real entry. Closing both tightly needs a per-tag entry-count or provenance check, tracked in #1887.

On a schedule, in your own repository:

1
name: Signer identity monitor
2
on:
3
  schedule:
4
    - cron: "17 * * * *"
5
  workflow_dispatch:
6
    inputs:
7
      bootstrap:
8
        description: "Start without a prior checkpoint, baselining at the current log head. Only for the very first run or a deliberate re-arm."
9
        type: boolean
10
        default: false
11
permissions: {}
12
concurrency:
13
  group: signer-identity-monitor
14
  cancel-in-progress: false
15
env:
16
  CERT_SUBJECT: '^https://github\.com/myorg/myrepo/\.github/workflows/release\.yaml@refs/tags/.*$'
17
  CERT_ISSUER: '^https://token\.actions\.githubusercontent\.com$'
18
  # Must name the same workflow as CERT_SUBJECT: the SAN is
19
  # <this-workflow>@refs/tags/<tag>, so this workflow's run history is the
20
  # authoritative record of which tags a real release signed. Change both together.
21
  RELEASE_WORKFLOW_FILE: release.yaml
22
jobs:
23
  monitor:
24
    runs-on: ubuntu-latest
25
    timeout-minutes: 90
26
    permissions:
27
      contents: read
28
      actions: read   # prior checkpoint artifact + release workflow run history
29
    steps:
30
      - uses: actions/checkout@v7
31
        with:
32
          repository: NVIDIA/aicr
33
          # Pin the tool you audit by commit SHA. A scheduled security job should
34
          # not float on another project's default branch, and a tag is mutable;
35
          # bump this deliberately after review.
36
          ref: d4f7bef460dc8d1ef7ea0334a6935c0038de88e4
37
      - uses: actions/setup-go@v7
38
        with:
39
          go-version-file: go.mod
40
41
      - name: Fetch previous checkpoint
42
        env:
43
          GH_TOKEN: ${{ github.token }}
44
          DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
45
          ALLOW_BOOTSTRAP: ${{ github.event_name == 'workflow_dispatch' && inputs.bootstrap }}
46
        run: |
47
          set -euo pipefail
48
          # The by-name artifact query is repo-wide: it also returns same-named
49
          # artifacts uploaded by a fork pull_request run, whose GITHUB_TOKEN can
50
          # upload under any name. A poisoned checkpoint (a genuine one copied
51
          # from the live log head) passes the consistency proof while collapsing
52
          # the identity-scan window to empty, silently skipping entries. So
53
          # accept only artifacts produced by this repository on its default
54
          # branch: head_repository_id == repository_id rejects every fork run,
55
          # and the head_branch match limits it to the scheduled runs.
56
          id="$(gh api --paginate \
57
            "repos/${GITHUB_REPOSITORY}/actions/artifacts?name=rekor-v2-checkpoint&per_page=100" \
58
            | jq -rs --arg branch "${DEFAULT_BRANCH}" '[.[].artifacts[]
59
                | select(.expired == false)
60
                | select(.workflow_run.head_branch == $branch)
61
                | select(.workflow_run.head_repository_id == .workflow_run.repository_id)]
62
                | sort_by(.created_at) | last | .id // empty')"
63
          if [ -n "${id}" ]; then
64
            gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${id}/zip" > checkpoint.zip
65
            exit 0
66
          fi
67
          # No usable cursor: never uploaded, or expired past retention-days.
68
          # Baselining here would silently skip every entry added since the last
69
          # good run, so bootstrapping has to be asked for. A scheduled run fails
70
          # as an operational error instead.
71
          if [ "${ALLOW_BOOTSTRAP}" = "true" ]; then
72
            echo "Bootstrap requested: starting without a checkpoint."
73
            exit 0
74
          fi
75
          echo "::error::No usable checkpoint artifact from this repository on ${DEFAULT_BRANCH}."
76
          echo "Re-arm deliberately with a workflow_dispatch run and bootstrap=true."
77
          exit 1
78
79
      - name: Fetch signed release tags
80
        env:
81
          GH_TOKEN: ${{ github.token }}
82
        run: |
83
          set -euo pipefail
84
          # The suppression allowlist must come from an authoritative record of
85
          # what actually signed, NOT from `git tag --list`: an attacker-pushed or
86
          # never-signed tag sitting in the repository would be pre-suppressed.
87
          # The SAN is <workflow>@refs/tags/<tag>, so a tag-push run of
88
          # RELEASE_WORKFLOW_FILE is the proof that a real release signed <tag>.
89
          # Run history also persists after a tag or release is deleted, so an
90
          # ephemeral release candidate stays recognised. head_branch is the tag
91
          # for a tag-push run. Require status=completed but accept any
92
          # conclusion: signing happens mid-run, so a release that flaked in a
93
          # later step still produced a legitimate entry.
94
          gh api --paginate \
95
            "repos/${GITHUB_REPOSITORY}/actions/workflows/${RELEASE_WORKFLOW_FILE}/runs?event=push&status=completed&per_page=100" \
96
            > runs.json
97
          jq -rs '[.[].workflow_runs[] | select(.head_branch != null) | .head_branch] | unique | .[]' \
98
            runs.json > known-tags.txt
99
          echo "Allowlisted $(wc -l < known-tags.txt) signed release tags."
100
101
      - name: Scan for our signing identity
102
        run: |
103
          # Without --known-tags-file every legitimate release alerts, and the
104
          # tool holds the cursor on a match, so the first real release after
105
          # baseline would suspend the scan until triaged.
106
          GOFLAGS="-mod=vendor" go run ./tools/rekor-monitor \
107
            --file checkpoint_v2.txt \
108
            --restore-zip checkpoint.zip \
109
            --cert-subject "${CERT_SUBJECT}" \
110
            --cert-issuer "${CERT_ISSUER}" \
111
            --known-tags-file known-tags.txt
112
113
      - uses: actions/upload-artifact@v7
114
        if: ${{ !cancelled() }}
115
        with:
116
          name: rekor-v2-checkpoint
117
          # The cursor plus its .scan and .stall companions. All three must
118
          # travel together so the next run resumes an in-progress catch-up.
119
          path: |
120
            checkpoint_v2.txt
121
            checkpoint_v2.txt.scan
122
            checkpoint_v2.txt.stall
123
          # The cursor has to outlast the longest gap between *successful* runs,
124
          # which is a much wider window than the cron interval: an expired
125
          # artifact is a lost cursor. The ceiling is the repository's artifact
126
          # retention maximum, 90 days on a public repository.
127
          retention-days: 30
128
          if-no-files-found: ignore

The ref: above and the git checkout --detach in the shell example are already full-length commit SHAs, because the source they pull in is the monitor you are trusting and a tag is mutable. The uses: action references are left as readable version tags so the example stays legible; pin those by SHA too in production, the same argument the digest-pinning advice earlier in this guide makes for images.

Two further hardening steps from AICR’s own workflow are worth copying before you rely on this. It branches notifications on the CLASSIFICATION= value, so that Sigstore or GitHub-API flakiness produces a quiet, self-healing failure rather than paging like a security event. And it wraps every checkpoint API call in a retry helper (.github/scripts/gh-api-retry.sh), which recovers 5xx and 429 responses and publishes each response through OUTFILE.part so a failed attempt’s error body can never be mistaken for a checkpoint. The example above is left plain because it already fails closed.

Monitoring Rekor v1 and a private Rekor

If you opted signing out to Rekor v1 with aicr bundle --rekor-url, be aware that neither of the two obvious cases has a turnkey monitor today. Both are coverage gaps, not recipes.

The public-good Rekor v1 gives no sustained coverage. Identity monitoring is a linear scan of every entry added since the last checkpoint, because Rekor’s index cannot be queried by certificate SAN. On the v1 firehose that scan runs roughly fifty times slower than the log grows, so a bounded CI job can never catch up. AICR measured exactly this before moving its own monitoring to v2 (see the rationale in .github/workflows/rekor-monitor.yaml). The upstream workflow will still complete its consistency check and record a baseline, but the identity search behind it falls further behind on every run.

A private Rekor v1 fails before it starts. Upstream resolves the checkpoint signing key by matching the log’s key ID against the Rekor logs listed in Sigstore’s TUF-distributed trusted root, which does not contain a private log’s key: GetLogVerifier returns couldn't find matching log instance and the run aborts. The upstream binary does accept --tuf-repository and --tuf-root-path to point at your own trust material, but the reusable workflow’s inputs are file_issue, artifact_retention_days, once, config, and url only, so there is no way to pass them through it. Running the binary directly against a TUF repository that serves your log’s key is the path here, not the reusable workflow.

What the upstream workflow does still illustrate is the configuration format, which is worth reading even if you end up driving the binary yourself. Sigstore’s walkthrough of rekor-monitor covers the same ground in more depth, including the split between its consistency check and its identity search, and the key-fingerprint mode below.

Read the block below for the shape of monitoredValues only. It is not a turnkey recipe: against the public-good Rekor v1 the identity search cannot keep up, and it carries no url because pointing it at a private log fails at verifier setup for the reason given above.

1
name: Signer identity monitor (Rekor v1)
2
on:
3
  schedule:
4
    - cron: "0 * * * *"
5
permissions: {}
6
jobs:
7
  monitor:
8
    permissions:
9
      contents: read
10
      issues: write
11
      # Required: upstream's detect-workflow job mints an OIDC token to resolve
12
      # which reusable repository and ref it is running from. Because that token
13
      # is minted as YOUR repository, the third-party workflow below must be
14
      # pinned by full-length commit SHA, not a mutable @main.
15
      id-token: write
16
    uses: sigstore/rekor-monitor/.github/workflows/reusable_monitoring.yml@aa97a44631e9ae0f2b262a77ee5ea8edbb11e26c
17
    with:
18
      file_issue: true
19
      artifact_retention_days: 14
20
      # No `url`: it defaults to the public-good Rekor v1, and a private log
21
      # cannot be reached through this workflow at all. Drive the binary
22
      # directly with --tuf-repository / --tuf-root-path instead.
23
      config: |
24
        monitoredValues:
25
          certIdentities:
26
            - certSubject: ^ci@myorg\.example\.com$
27
              issuers:
28
                - ^https://keycloak\.internal\.example\.com/realms/myorg$

issuers is matched against the certificate’s OIDC issuer extension (1.3.6.1.4.1.57264.1.8), which records the identity provider that authenticated the signer. It is not the Fulcio CA that issued the certificate, so a private Fulcio URL here matches nothing: put your own IdP’s issuer URL in it, as above. Upstream compiles both certSubject and each entry of issuers as Go regular expressions and requires both to match, so anchor them for the same reason you anchor the flags above, and keep the pair coherent the same way.

A KMS-signed entry carries no certificate, so there is no subject or issuer to match. Watch the key instead, via fingerprints, which is the hex-encoded SHA-256 of the DER-encoded public key:

$
cosign public-key --key gcpkms://projects/p/locations/l/keyRings/r/cryptoKeys/k \
>
  | openssl pkey -pubin -outform DER \
>
  | openssl dgst -sha256 -hex | awk '{print $NF}'

Then substitute that digest for the config: block in the workflow above:

1
      config: |
2
        monitoredValues:
3
          fingerprints:
4
            - <hex digest from the command above>

The remaining coverage gaps, stated plainly. Alongside the two Rekor v1 cases above, three more signing modes have no turnkey monitor:

• A KMS key signing to the default Rekor v2: tools/rekor-monitor exposes only the certificate identity flags, even though the upstream library it builds on does match v2 entries by key fingerprint.
• A private Rekor v2 (reached with aicr bundle --signing-config): out of reach for both tools, because tools/rekor-monitor resolves its shards from Sigstore’s TUF-distributed v2 signing config and takes no override.
• A private Fulcio signing into the public-good Rekor v2, the “Private Fulcio alone” row of the table (aicr bundle --fulcio-url with no --rekor-url): also unmonitorable by either tool, and this one fails quietly. tools/rekor-monitor materializes its trusted CA roots from Sigstore’s public TUF trusted root and offers no flag to add your own, so when the upstream identity search reaches an entry whose certificate chain does not validate against those roots, it writes a note to stderr and skips the entry. That is neither a match nor a failure, so entries under your private CA never reach the SAN and issuer comparison and the monitor stays green. Upstream’s binary does take --ca-roots and --ca-intermediates, but its reusable workflow exposes no input for them and cannot read Rekor v2 in the first place.

In every case, treat tools/rekor-monitor as a small reference implementation to adapt, or sign those artifacts to a target one of the two monitors already covers.

Responding to a hit

1. Rule yourself out first. Cross-check the entry’s log index and integrated timestamp against your own CI run history and any local signing you did. AICR automates this for its release identity by correlating matches against its release workflow’s run history; for a personal identity the equivalent is asking whether you signed anything at that moment.

2. Confirm the match is really your identity. Re-read the entry’s certificate SAN and OIDC issuer, or its key fingerprint, against what you configured. An unanchored regex is the usual cause of a false positive.

3. Treat an unexplained hit as compromise of the identity, not of the artifact. What “contain it” means depends on the signing mode:

   • Interactive keyless, an email in the SAN. The OIDC account is what was taken. Rotate its credentials, revoke active sessions and tokens, and read the identity provider’s own audit log.
   • CI workload identity, a workflow URL in the SAN. There is no account session to rotate here. The SAN names a workflow inside a repository, so containment means treating that repository as compromised: audit and narrow the workflow permissions, rotate every secret and deployment token those workflows can reach, review the environments and their protection rules, and look through the named workflow’s run history for a run nobody triggered.
   • KMS key. Rotate the key and revoke the principals allowed to sign with it.

4. Re-verify what you already published, but do not mistake what that proves. The command depends on what you signed:

   • Keyless bundles: aicr verify ./my-bundle --require-creator <identity>.
   • KMS-signed bundles: there is no certificate creator to require, so verify against the key itself with aicr verify ./my-bundle --key <kms-uri-or-pem>.
   • Recipe evidence: aicr evidence verify <evidence-bundle>, pinning the signer with --expected-issuer and --expected-identity-regexp.
   • Anything signed against a private Sigstore: add --trust-root ./trusted_root.json to aicr verify.

   None of these separates yours from the attacker’s inside the matched set, because a compromised identity satisfies the check exactly as well as you do. Establish legitimacy from a record the attacker does not control: compare each artifact’s digest and provenance against your own build and release history. Quarantine anything you cannot account for there.

5. Expect the entry to be permanent. Nothing can be removed from a transparency log, so the response is rotation plus a public statement of which entries are legitimate, never takedown.

References

• GitHub Artifact Attestations
• SLSA Framework
• GitHub Actions SLSA Generation
• SPDX Specification
• Sigstore Cosign
• Sigstore Policy Controller
• Kyverno Image Verification
• Sigstore Rekor, on why artifact owners should monitor the log for their own identity
• Sigstore Threat Model, which positions identity and consistency monitoring as the detection control for a compromised signer
• Using rekor-monitor to Scan Your Transparency Logs, Sigstore’s walkthrough of consistency checking, identity search, and key-fingerprint monitoring
• sigstore/rekor-monitor, the upstream tool and its reusable workflow inputs
• sigstore.dev and Rekor evolution, Sigstore’s current statement on Rekor v1 remaining the public-good default and v2 being opt-in