Known Issues#

The following issues apply to NVIDIA GPU Operator 26.7.x on Red Hat OpenShift Container Platform.

oc-mirror fails to mirror the GPU Operator bundle to Red Hat Quay#

When oc-mirror mirrors the GPU Operator catalog to a Red Hat Quay registry, copying two images in the Operator bundle fails with the following error:

[ERROR] : [Worker] error mirroring image docker://nvcr.io/nvidia/cloud-native/<image>@sha256:...
error: Manifest list must be converted to type "application/vnd.docker.distribution.manifest.list.v2+json"
to be written to destination, but we cannot modify it: "Instructed to preserve digests"

oc-mirror then omits the GPU Operator bundle from the mirror because one of the related images in the bundle failed to mirror. The mirrored catalog does not contain an installable GPU Operator.

Two different manifest inconsistencies cause this failure:

  • dra-driver-nvidia-gpu uses the Docker Schema2 manifest format with an uncompressed layer. The Docker Schema2 specification defines no uncompressed layer media type, so every Red Hat Quay version rejects it.

  • gdrdrv’s OCI image index references Docker Schema2 child manifests. Red Hat Quay releases before 3.16.7 and 3.17.0 reject this combination. Red Hat Quay 3.17.0 introduced a fix for this as PROJQUAY-11689, documented in the Red Hat Quay 3.17 release notes. Red Hat Quay 3.16.7 backports the same fix as PROJQUAY-11694. No earlier release line, including 3.15 and 3.14, received this backport.

Clusters that use the DRA Driver for NVIDIA GPUs have no workaround, because every Red Hat Quay version rejects the uncompressed layer in dra-driver-nvidia-gpu. The procedure on this page blocks the DRA driver image, so the DRA driver pods cannot start in the disconnected cluster.

Affected images#

The GPU Operator bundle lists the following images that fail to mirror to Red Hat Quay:

  • nvcr.io/nvidia/dra-driver-nvidia-gpu:v0.5.1

  • nvcr.io/nvidia/cloud-native/gdrdrv:v2.6-rhel9.6

Mirror the Operator bundle#

The procedure blocks the two affected images so that oc-mirror mirrors the rest of the GPU Operator bundle.

  1. Add the blockedImages entries to the mirror section of the ImageSetConfiguration file:

    apiVersion: mirror.openshift.io/v2alpha1
    kind: ImageSetConfiguration
    mirror:
      operators:
      - catalog: registry.redhat.io/redhat/certified-operator-index:v<ocp-version>
        packages:
        - name: gpu-operator-certified
      blockedImages:
      - name: 'nvcr[.]io/nvidia/dra-driver-nvidia-gpu.*@sha256:ae591dbc8e630b49181b40dfddf8f0a3f0890b883acb7b2b420f5b207fda253e'
      - name: 'nvcr[.]io/nvidia/cloud-native/gdrdrv@sha256:eef1b6663d78555f0e5c49c156196b67e8a3008bc22aaa02b03dad53887ee0f1'
    

    Tip

    If the mirror registry runs Red Hat Quay 3.16.7, 3.17.0, or later, remove the gdrdrv entry from blockedImages.

  2. Rerun the oc-mirror command with the updated ImageSetConfiguration file.

  3. Apply the generated ImageDigestMirrorSet and CatalogSource resources from the working-dir/cluster-resources directory:

    $ oc apply -f <workspace>/working-dir/cluster-resources/idms-oc-mirror.yaml
    $ oc apply -f <workspace>/working-dir/cluster-resources/cs-<catalog>.yaml
    
  4. Install the GPU Operator from the mirrored catalog.

Check an image for manifest problems#

To check an image that fails to mirror, use skopeo to inspect the image manifests.

  1. Display the media type of the top-level manifest:

    $ skopeo inspect --raw docker://<image-reference> | jq -r .mediaType
    
  2. If the media type is application/vnd.oci.image.index.v1+json, display the media types of the per-architecture manifests:

    $ skopeo inspect --raw docker://<image-reference> | jq '.manifests[] | {platform, mediaType, digest}'
    

    A per-architecture media type of application/vnd.docker.distribution.manifest.v2+json does not match the OCI image index.

  3. Display the layer media types of a per-architecture manifest:

    $ skopeo inspect --raw docker://<image-reference>@<manifest-digest> | jq -r '.layers[].mediaType'
    

    A layer media type that does not end in gzip or zstd, such as application/vnd.oci.image.layer.v1.tar, identifies an uncompressed layer.

    An uncompressed layer under an OCI image manifest, application/vnd.oci.image.layer.v1.tar, mirrors without a problem. The OCI Image Specification defines this media type. An uncompressed layer under a Docker Schema2 manifest, application/vnd.docker.image.rootfs.diff.tar, does not mirror. The Docker Image Manifest V2, Schema 2 specification defines no uncompressed layer media type, so registries reject it.

To report an affected image that this page does not list, add the image reference and the oc-mirror error output to NVIDIA/gpu-operator#3166.