You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Feature]: Support gds.enabled=true with precompiled drivers when nvidia_fs is pre-built into the driver image #3165
When using precompiled NVIDIA drivers, gds.enabled=true is currently hard-rejected in transformGDSContainer() with the error:
GPUDirect Storage driver (nvidia-fs) is not supported along with pre-compiled NVIDIA drivers
The reason is that the nvidia-fs-ctr sidecar waits for /run/nvidia/driver/usr/src to be
populated, then runs nvidia-gds-driver install which builds nvidia_fs.ko from those sources.
With precompiled drivers there are no sources at that path, so the install would fail.
This assumption breaks down when nvidia_fs.ko is pre-built into the precompiled driver image.
In that case the module is already loaded by the time nvidia-fs-ctr starts, making the build
step in nvidia-gds-driver install unnecessary.
The request is to support a load-only mode in transformGDSContainer() for this case: when UsePrecompiledDrivers() is true and nvidia_fs is pre-built into the image, skip the nvidia-gds-driver install step and only run the GDS toolkit setup (verify module is loaded,
write cufile.json, configure device permissions) rather than returning an error.
We have the same use case and tested it on GPU Operator v26.7.3.
Our precompiled driver image already contains nvidia-fs.ko and gdrdrv.ko. Since gds.enabled and gdrcopy.enabled are refused with usePrecompiled: true, our driver container loads the modules and creates their device nodes when GDS_ENABLED=true or GDRCOPY_ENABLED=true is set in the NVIDIADriverspec.env.
Everything after the module load worked unchanged, so the load step is the only missing part.
Could the operator allow gds.enabled with usePrecompiled: true when the driver image already carries the module? As far as we know, relying on spec.env passing through isn't documented.
Activity
aamlani2 commentedon Nov 21, 2026
We have the same use case and tested it on GPU Operator v26.7.3.
Our precompiled driver image already contains
nvidia-fs.koandgdrdrv.ko. Sincegds.enabledandgdrcopy.enabledare refused withusePrecompiled: true, our driver container loads the modules and creates their device nodes whenGDS_ENABLED=trueorGDRCOPY_ENABLED=trueis set in theNVIDIADriverspec.env.Everything after the module load worked unchanged, so the load step is the only missing part.
Could the operator allow
gds.enabledwithusePrecompiled: truewhen the driver image already carries the module? As far as we know, relying onspec.envpassing through isn't documented.