diff --git a/openshift/index.rst b/openshift/index.rst index 2f072a365..c39e233c8 100644 --- a/openshift/index.rst +++ b/openshift/index.rst @@ -16,6 +16,7 @@ NVIDIA GPU Operator on Red Hat OpenShift Container Platform clean-up.rst mirror-gpu-ocp-disconnected.rst deploy-gpu-ocp-ipv6.rst + known-issues.rst enable-gpu-monitoring-dashboard.rst time-slicing-gpus-in-openshift.rst openshift-virtualization.rst diff --git a/openshift/known-issues.rst b/openshift/known-issues.rst new file mode 100644 index 000000000..41d733920 --- /dev/null +++ b/openshift/known-issues.rst @@ -0,0 +1,120 @@ +.. Date: November 18 2026 +.. Author: nbugrovs + +.. headings are # * - = + +.. _openshift-known-issues: + +##################################### +Known Issues +##################################### + +The following issues apply to NVIDIA GPU Operator 26.7.x on Red Hat OpenShift Container Platform. + +.. _openshift-known-issues-oc-mirror-quay: + +********************************************************************* +``oc-mirror`` fails to mirror the GPU Operator bundle to Red Hat Quay +********************************************************************* + +When ``oc-mirror`` mirrors the GPU Operator catalog to a Red Hat Quay registry, copying two images in the Operator bundle fails with the following error: + +.. code-block:: output + + [ERROR] : [Worker] error mirroring image docker://nvcr.io/nvidia/cloud-native/@sha256:... + error: Manifest list must be converted to type "application/vnd.docker.distribution.manifest.list.v2+json" + to be written to destination, but we cannot modify it: "Instructed to preserve digests" + +``oc-mirror`` then omits the GPU Operator bundle from the mirror because one of the related images in the bundle failed to mirror. +The mirrored catalog does not contain an installable GPU Operator. + +Two different manifest inconsistencies cause this failure: + +* ``dra-driver-nvidia-gpu`` uses the Docker Schema2 manifest format with an uncompressed layer. + The Docker Schema2 specification defines no uncompressed layer media type, so every Red Hat Quay version rejects it. +* ``gdrdrv``'s OCI image index references Docker Schema2 child manifests. + Red Hat Quay releases before 3.16.7 and 3.17.0 reject this combination. + Red Hat Quay 3.17.0 introduced a fix for this as `PROJQUAY-11689 `_, documented in the `Red Hat Quay 3.17 release notes `_. + Red Hat Quay 3.16.7 backports the same fix as `PROJQUAY-11694 `_. + No earlier release line, including 3.15 and 3.14, received this backport. + +Clusters that use the DRA Driver for NVIDIA GPUs have no workaround, because every Red Hat Quay version rejects the uncompressed layer in ``dra-driver-nvidia-gpu``. +The procedure on this page blocks the DRA driver image, so the DRA driver pods cannot start in the disconnected cluster. + +Affected images +=============== + +The GPU Operator bundle lists the following images that fail to mirror to Red Hat Quay: + +* ``nvcr.io/nvidia/dra-driver-nvidia-gpu:v0.5.1`` +* ``nvcr.io/nvidia/cloud-native/gdrdrv:v2.6-rhel9.6`` + +Mirror the Operator bundle +========================== + +The procedure blocks the two affected images so that ``oc-mirror`` mirrors the rest of the GPU Operator bundle. + +#. Add the ``blockedImages`` entries to the ``mirror`` section of the ``ImageSetConfiguration`` file: + + .. code-block:: yaml + :emphasize-lines: 8-10 + + apiVersion: mirror.openshift.io/v2alpha1 + kind: ImageSetConfiguration + mirror: + operators: + - catalog: registry.redhat.io/redhat/certified-operator-index:v + packages: + - name: gpu-operator-certified + blockedImages: + - name: 'nvcr[.]io/nvidia/dra-driver-nvidia-gpu.*@sha256:ae591dbc8e630b49181b40dfddf8f0a3f0890b883acb7b2b420f5b207fda253e' + - name: 'nvcr[.]io/nvidia/cloud-native/gdrdrv@sha256:eef1b6663d78555f0e5c49c156196b67e8a3008bc22aaa02b03dad53887ee0f1' + + .. tip:: + + If the mirror registry runs Red Hat Quay 3.16.7, 3.17.0, or later, remove the ``gdrdrv`` entry from ``blockedImages``. + +#. Rerun the ``oc-mirror`` command with the updated ``ImageSetConfiguration`` file. + +#. Apply the generated ``ImageDigestMirrorSet`` and ``CatalogSource`` resources from the ``working-dir/cluster-resources`` directory: + + .. code-block:: console + + $ oc apply -f /working-dir/cluster-resources/idms-oc-mirror.yaml + $ oc apply -f /working-dir/cluster-resources/cs-.yaml + +#. Install the GPU Operator from the mirrored catalog. + +Check an image for manifest problems +==================================== + +To check an image that fails to mirror, use ``skopeo`` to inspect the image manifests. + +#. Display the media type of the top-level manifest: + + .. code-block:: console + + $ skopeo inspect --raw docker:// | jq -r .mediaType + +#. If the media type is ``application/vnd.oci.image.index.v1+json``, display the media types of the per-architecture manifests: + + .. code-block:: console + + $ skopeo inspect --raw docker:// | jq '.manifests[] | {platform, mediaType, digest}' + + A per-architecture media type of ``application/vnd.docker.distribution.manifest.v2+json`` does not match the OCI image index. + +#. Display the layer media types of a per-architecture manifest: + + .. code-block:: console + + $ skopeo inspect --raw docker://@ | jq -r '.layers[].mediaType' + + A layer media type that does not end in ``gzip`` or ``zstd``, such as ``application/vnd.oci.image.layer.v1.tar``, identifies an uncompressed layer. + + An uncompressed layer under an OCI image manifest, ``application/vnd.oci.image.layer.v1.tar``, mirrors without a problem. + The OCI Image Specification defines this media type. + An uncompressed layer under a Docker Schema2 manifest, ``application/vnd.docker.image.rootfs.diff.tar``, does not mirror. + The Docker Image Manifest V2, Schema 2 specification defines no uncompressed layer media type, so registries reject it. + +To report an affected image that this page does not list, add the image reference and the ``oc-mirror`` error output to `NVIDIA/gpu-operator#3166 `_. diff --git a/openshift/mirror-gpu-ocp-disconnected.rst b/openshift/mirror-gpu-ocp-disconnected.rst index d2e307bcb..0dfac8417 100644 --- a/openshift/mirror-gpu-ocp-disconnected.rst +++ b/openshift/mirror-gpu-ocp-disconnected.rst @@ -656,6 +656,11 @@ You can mirror the Operator content of a Red Hat-provided catalog, or a custom c container image registry using the ``oc-mirror`` commands. The target registry must support `Docker v2-2 `_. For a cluster on a restricted network, this registry can be one that the cluster has network access to, such as a mirror registry created during a restricted network cluster installation. +.. note:: + + Some GPU Operator related images have known container image manifest issues that can cause mirroring to fail. + Refer to :doc:`known-issues` for how to identify an affected image and for the current workaround. + The ``oc-mirror`` commands also automatically mirrors the index image specified during the mirroring process, whether it be a Red Hat-provided index image or your own custom-built index image, to the target registry. diff --git a/openshift/troubleshooting-gpu-ocp.rst b/openshift/troubleshooting-gpu-ocp.rst index 3049f75ed..87bebfb8f 100644 --- a/openshift/troubleshooting-gpu-ocp.rst +++ b/openshift/troubleshooting-gpu-ocp.rst @@ -9,6 +9,10 @@ Troubleshooting This section includes errors that users might encounter when performing various checks during installation of the **NVIDIA GPU Operator** on the OpenShift Container Platform cluster. +.. note:: + + If you deploy the GPU Operator in a disconnected or airgapped environment and mirroring an image fails, refer to :doc:`known-issues`. + Node Feature Discovery checks -----------------------------------