Skip to content

[Feature]: Support gds.enabled=true with precompiled drivers when nvidia_fs is pre-built into the driver image #3165

Open

Description

@lizzzcai

When using precompiled NVIDIA drivers, gds.enabled=true is currently hard-rejected in
transformGDSContainer() with the error:

GPUDirect Storage driver (nvidia-fs) is not supported along with pre-compiled NVIDIA drivers

The reason is that the nvidia-fs-ctr sidecar waits for /run/nvidia/driver/usr/src to be
populated, then runs nvidia-gds-driver install which builds nvidia_fs.ko from those sources.
With precompiled drivers there are no sources at that path, so the install would fail.

This assumption breaks down when nvidia_fs.ko is pre-built into the precompiled driver image.
In that case the module is already loaded by the time nvidia-fs-ctr starts, making the build
step in nvidia-gds-driver install unnecessary.

The request is to support a load-only mode in transformGDSContainer() for this case: when
UsePrecompiledDrivers() is true and nvidia_fs is pre-built into the image, skip the
nvidia-gds-driver install step and only run the GDS toolkit setup (verify module is loaded,
write cufile.json, configure device permissions) rather than returning an error.

Activity

  1. added
    featureissue/PR that proposes a new feature or functionality
    needs-triageissue or PR has not been assigned a priority-px label
    on Nov 3, 2026
  2. aamlani2 commented on Nov 21, 2026

    @aamlani2

    We have the same use case and tested it on GPU Operator v26.7.3.

    Our precompiled driver image already contains nvidia-fs.ko and gdrdrv.ko. Since gds.enabled and gdrcopy.enabled are refused with usePrecompiled: true, our driver container loads the modules and creates their device nodes when GDS_ENABLED=true or GDRCOPY_ENABLED=true is set in the NVIDIADriver spec.env.

    Everything after the module load worked unchanged, so the load step is the only missing part.
    Could the operator allow gds.enabled with usePrecompiled: true when the driver image already carries the module? As far as we know, relying on spec.env passing through isn't documented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureissue/PR that proposes a new feature or functionalitylifecycle/frozenneeds-triageissue or PR has not been assigned a priority-px label

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions