Skip to content

Fix undersized reduction workspac - #115

Open
LucasBoTang wants to merge 1 commit into
MIT-Lu-Lab:mainfrom
LucasBoTang:fix/reduction-workspace
Open

LucasBoTang wants to merge 1 commit into
MIT-Lu-Lab:mainfrom
LucasBoTang:fix/reduction-workspace

Conversation

@LucasBoTang

Copy link
Copy Markdown
Collaborator

On Windows with CUDA 13.4, test_time_limit fails inside cub::DeviceReduce::Max:

CUDA Error at .../internal/reduce.h:60: cudaErrorInvalidValue

Cause

This regression was introduced by #112, which added these CUB reductions and their shared workspace. The workspace size comes from asking CUB what a plain double * reduction over the longest vector needs. The real reductions use lambdas instead, and CUB's requirement depends on the kernel it builds for each lambda.

On the test_time_limit problem, the workspace is 13,823 bytes. Every reduction fits except the six second-round infeasibility maxima, |A||d| and |A^T||y| for three candidates, which need 20,479 bytes each. The second round runs only when a candidate passes its objective-margin test, so most solves never reach it.

Fix

Each group of reductions now takes a required_bytes argument. With NULL it runs as before. With a pointer it only asks CUB for its size and raises *required_bytes to it.

At startup the solver queries every group this way, including those that may never run, and then allocates the workspace once. Planning and execution call the same functions, so CUB sees the same lambdas.

This PR also adds test_reduce_workspace, and CMake now builds .cu tests as HIP sources when USE_HIP is on.

Reproduction

Windows 11, MSVC 19.44, conda-forge CUDA 13.4.92, RTX 5080. Run this in an x64 MSVC developer shell with the conda environment active:

git clone https://github.com/MIT-Lu-Lab/cuPDLPx.git cupdlpx-repro
cd cupdlpx-repro
git checkout 724894e

$env:CUDA_PATH = Join-Path $env:CONDA_PREFIX "Library"
$env:CUDACXX = Join-Path $env:CUDA_PATH "bin/nvcc.exe"
$env:CMAKE_GENERATOR = "Ninja"
$env:CMAKE_GENERATOR_PLATFORM = ""
$env:CMAKE_GENERATOR_TOOLSET = ""
$env:CUDAFLAGS = "-Xcompiler=/Zc:preprocessor"
python -m pip install ".[test]" -Ccmake.define.CMAKE_CUDA_ARCHITECTURES=120

@'
import ctypes
import os
import pytest

cuda = ctypes.WinDLL(os.path.join(os.environ["CUDA_PATH"], "bin", "cudart64_13.dll"))
assert cuda.cudaFree(None) == 0
raise SystemExit(pytest.main(["test/test_limit.py::test_time_limit", "-s", "--no-cov"]))
'@ | python

/Zc:preprocessor is needed until #113 is merged. CUDA is initialized first so that the startup race fixed in #114 does not hide this error.

On 724894e the test fails with the error above. With this PR it passes.

Validation

  • Windows, CUDA 13.4: test_reduce_workspace, test_time_limit and all 107 Python tests passed, with CUDA initialized first.
  • Windows, CUDA 12.9: all 107 Python tests passed.
  • Windows, CUDA 12.4: test_reduce_workspace and test_interface passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant