From eadcc7eb550cd09bfa8659babe54877a4dbfef49 Mon Sep 17 00:00:00 2001 From: Nikita Bugrovsky Date: Thu, 19 Nov 2026 18:41:57 +0200 Subject: [PATCH] openshift: add known issues page for image manifest mirroring failures Add a Known Issues page under openshift/ documenting that oc-mirror fails to mirror two GPU Operator bundle images to Red Hat Quay: dra-driver-nvidia-gpu:v0.5.1 and the gdrdrv image pinned in the bundle. Both fail because of container image manifest defects, not because digest preservation is merely inconvenient. Findings, confirmed by direct skopeo and oc-mirror testing against live Red Hat Quay 3.16.2 and 3.17.3 instances: * gdrdrv's OCI image index references Docker Schema2 child manifests. Red Hat Quay rejects this combination before 3.17, and accepts it unchanged on 3.17 and later. The Red Hat Quay 3.17 release notes document this as a fixed limitation. * dra-driver-nvidia-gpu uses the Docker Schema2 manifest format with an uncompressed layer. The Docker Schema2 specification defines no uncompressed layer media type, so every Red Hat Quay version rejects it. Clusters that use the DRA Driver for NVIDIA GPUs have no workaround. * The default driver manager image, k8s-driver-manager:v0.12.1, also has an uncompressed layer, but its OCI manifest format permits uncompressed layers, so it mirrors to Red Hat Quay unchanged. Confirmed with a digest-identical mirror to a live Quay 3.16.2 instance. This image is excluded from the blockedImages workaround on this page. Includes an ImageSetConfiguration blockedImages procedure for the two affected images, a tip to drop the gdrdrv block on Red Hat Quay 3.17 and later, and skopeo commands to check any image for the same class of manifest problem. Cross-link the new page from the disconnected mirroring guide and the troubleshooting page. Refs: https://github.com/NVIDIA/gpu-operator/issues/3166 Signed-off-by: Nikita Bugrovsky --- openshift/index.rst | 1 + openshift/known-issues.rst | 120 ++++++++++++++++++++++ openshift/mirror-gpu-ocp-disconnected.rst | 5 + openshift/troubleshooting-gpu-ocp.rst | 4 + 4 files changed, 130 insertions(+) create mode 100644 openshift/known-issues.rst diff --git a/openshift/index.rst b/openshift/index.rst index 2f072a365..c39e233c8 100644 --- a/openshift/index.rst +++ b/openshift/index.rst @@ -16,6 +16,7 @@ NVIDIA GPU Operator on Red Hat OpenShift Container Platform clean-up.rst mirror-gpu-ocp-disconnected.rst deploy-gpu-ocp-ipv6.rst + known-issues.rst enable-gpu-monitoring-dashboard.rst time-slicing-gpus-in-openshift.rst openshift-virtualization.rst diff --git a/openshift/known-issues.rst b/openshift/known-issues.rst new file mode 100644 index 000000000..41d733920 --- /dev/null +++ b/openshift/known-issues.rst @@ -0,0 +1,120 @@ +.. Date: November 18 2026 +.. Author: nbugrovs + +.. headings are # * - = + +.. _openshift-known-issues: + +##################################### +Known Issues +##################################### + +The following issues apply to NVIDIA GPU Operator 26.7.x on Red Hat OpenShift Container Platform. + +.. _openshift-known-issues-oc-mirror-quay: + +********************************************************************* +``oc-mirror`` fails to mirror the GPU Operator bundle to Red Hat Quay +********************************************************************* + +When ``oc-mirror`` mirrors the GPU Operator catalog to a Red Hat Quay registry, copying two images in the Operator bundle fails with the following error: + +.. code-block:: output + + [ERROR] : [Worker] error mirroring image docker://nvcr.io/nvidia/cloud-native/@sha256:... + error: Manifest list must be converted to type "application/vnd.docker.distribution.manifest.list.v2+json" + to be written to destination, but we cannot modify it: "Instructed to preserve digests" + +``oc-mirror`` then omits the GPU Operator bundle from the mirror because one of the related images in the bundle failed to mirror. +The mirrored catalog does not contain an installable GPU Operator. + +Two different manifest inconsistencies cause this failure: + +* ``dra-driver-nvidia-gpu`` uses the Docker Schema2 manifest format with an uncompressed layer. + The Docker Schema2 specification defines no uncompressed layer media type, so every Red Hat Quay version rejects it. +* ``gdrdrv``'s OCI image index references Docker Schema2 child manifests. + Red Hat Quay releases before 3.16.7 and 3.17.0 reject this combination. + Red Hat Quay 3.17.0 introduced a fix for this as `PROJQUAY-11689 `_, documented in the `Red Hat Quay 3.17 release notes `_. + Red Hat Quay 3.16.7 backports the same fix as `PROJQUAY-11694 `_. + No earlier release line, including 3.15 and 3.14, received this backport. + +Clusters that use the DRA Driver for NVIDIA GPUs have no workaround, because every Red Hat Quay version rejects the uncompressed layer in ``dra-driver-nvidia-gpu``. +The procedure on this page blocks the DRA driver image, so the DRA driver pods cannot start in the disconnected cluster. + +Affected images +=============== + +The GPU Operator bundle lists the following images that fail to mirror to Red Hat Quay: + +* ``nvcr.io/nvidia/dra-driver-nvidia-gpu:v0.5.1`` +* ``nvcr.io/nvidia/cloud-native/gdrdrv:v2.6-rhel9.6`` + +Mirror the Operator bundle +========================== + +The procedure blocks the two affected images so that ``oc-mirror`` mirrors the rest of the GPU Operator bundle. + +#. Add the ``blockedImages`` entries to the ``mirror`` section of the ``ImageSetConfiguration`` file: + + .. code-block:: yaml + :emphasize-lines: 8-10 + + apiVersion: mirror.openshift.io/v2alpha1 + kind: ImageSetConfiguration + mirror: + operators: + - catalog: registry.redhat.io/redhat/certified-operator-index:v + packages: + - name: gpu-operator-certified + blockedImages: + - name: 'nvcr[.]io/nvidia/dra-driver-nvidia-gpu.*@sha256:ae591dbc8e630b49181b40dfddf8f0a3f0890b883acb7b2b420f5b207fda253e' + - name: 'nvcr[.]io/nvidia/cloud-native/gdrdrv@sha256:eef1b6663d78555f0e5c49c156196b67e8a3008bc22aaa02b03dad53887ee0f1' + + .. tip:: + + If the mirror registry runs Red Hat Quay 3.16.7, 3.17.0, or later, remove the ``gdrdrv`` entry from ``blockedImages``. + +#. Rerun the ``oc-mirror`` command with the updated ``ImageSetConfiguration`` file. + +#. Apply the generated ``ImageDigestMirrorSet`` and ``CatalogSource`` resources from the ``working-dir/cluster-resources`` directory: + + .. code-block:: console + + $ oc apply -f /working-dir/cluster-resources/idms-oc-mirror.yaml + $ oc apply -f /working-dir/cluster-resources/cs-.yaml + +#. Install the GPU Operator from the mirrored catalog. + +Check an image for manifest problems +==================================== + +To check an image that fails to mirror, use ``skopeo`` to inspect the image manifests. + +#. Display the media type of the top-level manifest: + + .. code-block:: console + + $ skopeo inspect --raw docker:// | jq -r .mediaType + +#. If the media type is ``application/vnd.oci.image.index.v1+json``, display the media types of the per-architecture manifests: + + .. code-block:: console + + $ skopeo inspect --raw docker:// | jq '.manifests[] | {platform, mediaType, digest}' + + A per-architecture media type of ``application/vnd.docker.distribution.manifest.v2+json`` does not match the OCI image index. + +#. Display the layer media types of a per-architecture manifest: + + .. code-block:: console + + $ skopeo inspect --raw docker://@ | jq -r '.layers[].mediaType' + + A layer media type that does not end in ``gzip`` or ``zstd``, such as ``application/vnd.oci.image.layer.v1.tar``, identifies an uncompressed layer. + + An uncompressed layer under an OCI image manifest, ``application/vnd.oci.image.layer.v1.tar``, mirrors without a problem. + The OCI Image Specification defines this media type. + An uncompressed layer under a Docker Schema2 manifest, ``application/vnd.docker.image.rootfs.diff.tar``, does not mirror. + The Docker Image Manifest V2, Schema 2 specification defines no uncompressed layer media type, so registries reject it. + +To report an affected image that this page does not list, add the image reference and the ``oc-mirror`` error output to `NVIDIA/gpu-operator#3166 `_. diff --git a/openshift/mirror-gpu-ocp-disconnected.rst b/openshift/mirror-gpu-ocp-disconnected.rst index d2e307bcb..0dfac8417 100644 --- a/openshift/mirror-gpu-ocp-disconnected.rst +++ b/openshift/mirror-gpu-ocp-disconnected.rst @@ -656,6 +656,11 @@ You can mirror the Operator content of a Red Hat-provided catalog, or a custom c container image registry using the ``oc-mirror`` commands. The target registry must support `Docker v2-2 `_. For a cluster on a restricted network, this registry can be one that the cluster has network access to, such as a mirror registry created during a restricted network cluster installation. +.. note:: + + Some GPU Operator related images have known container image manifest issues that can cause mirroring to fail. + Refer to :doc:`known-issues` for how to identify an affected image and for the current workaround. + The ``oc-mirror`` commands also automatically mirrors the index image specified during the mirroring process, whether it be a Red Hat-provided index image or your own custom-built index image, to the target registry. diff --git a/openshift/troubleshooting-gpu-ocp.rst b/openshift/troubleshooting-gpu-ocp.rst index 3049f75ed..87bebfb8f 100644 --- a/openshift/troubleshooting-gpu-ocp.rst +++ b/openshift/troubleshooting-gpu-ocp.rst @@ -9,6 +9,10 @@ Troubleshooting This section includes errors that users might encounter when performing various checks during installation of the **NVIDIA GPU Operator** on the OpenShift Container Platform cluster. +.. note:: + + If you deploy the GPU Operator in a disconnected or airgapped environment and mirroring an image fails, refer to :doc:`known-issues`. + Node Feature Discovery checks -----------------------------------