Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions openshift/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ NVIDIA GPU Operator on Red Hat OpenShift Container Platform
clean-up.rst
mirror-gpu-ocp-disconnected.rst
deploy-gpu-ocp-ipv6.rst
known-issues.rst
enable-gpu-monitoring-dashboard.rst
time-slicing-gpus-in-openshift.rst
openshift-virtualization.rst
Expand Down
120 changes: 120 additions & 0 deletions openshift/known-issues.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
.. Date: November 18 2026
.. Author: nbugrovs

.. headings are # * - =

.. _openshift-known-issues:

#####################################
Known Issues
#####################################

The following issues apply to NVIDIA GPU Operator 26.7.x on Red Hat OpenShift Container Platform.

.. _openshift-known-issues-oc-mirror-quay:

*********************************************************************
``oc-mirror`` fails to mirror the GPU Operator bundle to Red Hat Quay
*********************************************************************

When ``oc-mirror`` mirrors the GPU Operator catalog to a Red Hat Quay registry, copying two images in the Operator bundle fails with the following error:

.. code-block:: output

[ERROR] : [Worker] error mirroring image docker://nvcr.io/nvidia/cloud-native/<image>@sha256:...
error: Manifest list must be converted to type "application/vnd.docker.distribution.manifest.list.v2+json"
to be written to destination, but we cannot modify it: "Instructed to preserve digests"

``oc-mirror`` then omits the GPU Operator bundle from the mirror because one of the related images in the bundle failed to mirror.
The mirrored catalog does not contain an installable GPU Operator.

Two different manifest inconsistencies cause this failure:

* ``dra-driver-nvidia-gpu`` uses the Docker Schema2 manifest format with an uncompressed layer.
The Docker Schema2 specification defines no uncompressed layer media type, so every Red Hat Quay version rejects it.
* ``gdrdrv``'s OCI image index references Docker Schema2 child manifests.
Red Hat Quay releases before 3.16.7 and 3.17.0 reject this combination.
Red Hat Quay 3.17.0 introduced a fix for this as `PROJQUAY-11689 <https://issues.redhat.com/browse/PROJQUAY-11689>`_, documented in the `Red Hat Quay 3.17 release notes <https://docs.redhat.com/en/documentation/red_hat_quay/3.17/html/red_hat_quay_release_notes/release-notes-317>`_.
Red Hat Quay 3.16.7 backports the same fix as `PROJQUAY-11694 <https://issues.redhat.com/browse/PROJQUAY-11694>`_.
No earlier release line, including 3.15 and 3.14, received this backport.

Clusters that use the DRA Driver for NVIDIA GPUs have no workaround, because every Red Hat Quay version rejects the uncompressed layer in ``dra-driver-nvidia-gpu``.
The procedure on this page blocks the DRA driver image, so the DRA driver pods cannot start in the disconnected cluster.

Affected images
===============

The GPU Operator bundle lists the following images that fail to mirror to Red Hat Quay:

* ``nvcr.io/nvidia/dra-driver-nvidia-gpu:v0.5.1``
* ``nvcr.io/nvidia/cloud-native/gdrdrv:v2.6-rhel9.6``

Mirror the Operator bundle
==========================

The procedure blocks the two affected images so that ``oc-mirror`` mirrors the rest of the GPU Operator bundle.

#. Add the ``blockedImages`` entries to the ``mirror`` section of the ``ImageSetConfiguration`` file:

.. code-block:: yaml
:emphasize-lines: 8-10

apiVersion: mirror.openshift.io/v2alpha1
kind: ImageSetConfiguration
mirror:
operators:
- catalog: registry.redhat.io/redhat/certified-operator-index:v<ocp-version>
packages:
- name: gpu-operator-certified
blockedImages:
- name: 'nvcr[.]io/nvidia/dra-driver-nvidia-gpu.*@sha256:ae591dbc8e630b49181b40dfddf8f0a3f0890b883acb7b2b420f5b207fda253e'
- name: 'nvcr[.]io/nvidia/cloud-native/gdrdrv@sha256:eef1b6663d78555f0e5c49c156196b67e8a3008bc22aaa02b03dad53887ee0f1'

.. tip::

If the mirror registry runs Red Hat Quay 3.16.7, 3.17.0, or later, remove the ``gdrdrv`` entry from ``blockedImages``.

#. Rerun the ``oc-mirror`` command with the updated ``ImageSetConfiguration`` file.

#. Apply the generated ``ImageDigestMirrorSet`` and ``CatalogSource`` resources from the ``working-dir/cluster-resources`` directory:

.. code-block:: console

$ oc apply -f <workspace>/working-dir/cluster-resources/idms-oc-mirror.yaml
$ oc apply -f <workspace>/working-dir/cluster-resources/cs-<catalog>.yaml

#. Install the GPU Operator from the mirrored catalog.

Check an image for manifest problems
====================================

To check an image that fails to mirror, use ``skopeo`` to inspect the image manifests.

#. Display the media type of the top-level manifest:

.. code-block:: console

$ skopeo inspect --raw docker://<image-reference> | jq -r .mediaType

#. If the media type is ``application/vnd.oci.image.index.v1+json``, display the media types of the per-architecture manifests:

.. code-block:: console

$ skopeo inspect --raw docker://<image-reference> | jq '.manifests[] | {platform, mediaType, digest}'

A per-architecture media type of ``application/vnd.docker.distribution.manifest.v2+json`` does not match the OCI image index.

#. Display the layer media types of a per-architecture manifest:

.. code-block:: console

$ skopeo inspect --raw docker://<image-reference>@<manifest-digest> | jq -r '.layers[].mediaType'

A layer media type that does not end in ``gzip`` or ``zstd``, such as ``application/vnd.oci.image.layer.v1.tar``, identifies an uncompressed layer.

An uncompressed layer under an OCI image manifest, ``application/vnd.oci.image.layer.v1.tar``, mirrors without a problem.
The OCI Image Specification defines this media type.
An uncompressed layer under a Docker Schema2 manifest, ``application/vnd.docker.image.rootfs.diff.tar``, does not mirror.
The Docker Image Manifest V2, Schema 2 specification defines no uncompressed layer media type, so registries reject it.

To report an affected image that this page does not list, add the image reference and the ``oc-mirror`` error output to `NVIDIA/gpu-operator#3166 <https://github.com/NVIDIA/gpu-operator/issues/3166>`_.
5 changes: 5 additions & 0 deletions openshift/mirror-gpu-ocp-disconnected.rst
Original file line number Diff line number Diff line change
Expand Up @@ -656,6 +656,11 @@ You can mirror the Operator content of a Red Hat-provided catalog, or a custom c
container image registry using the ``oc-mirror`` commands.
The target registry must support `Docker v2-2 <https://docs.docker.com/registry/spec/manifest-v2-2/>`_. For a cluster on a restricted network, this registry can be one that the cluster has network access to, such as a mirror registry created during a restricted network cluster installation.

.. note::

Some GPU Operator related images have known container image manifest issues that can cause mirroring to fail.
Refer to :doc:`known-issues` for how to identify an affected image and for the current workaround.

The ``oc-mirror`` commands also automatically mirrors the index
image specified during the mirroring process, whether it be a Red Hat-provided index image or
your own custom-built index image, to the target registry.
Expand Down
4 changes: 4 additions & 0 deletions openshift/troubleshooting-gpu-ocp.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ Troubleshooting

This section includes errors that users might encounter when performing various checks during installation of the **NVIDIA GPU Operator** on the OpenShift Container Platform cluster.

.. note::

If you deploy the GPU Operator in a disconnected or airgapped environment and mirroring an image fails, refer to :doc:`known-issues`.

Node Feature Discovery checks
-----------------------------------

Expand Down