Repository navigation
Conversation
…on spaces The thicknesses of a SubDimension or MultiSubDimension are runtime values, so a device parallel loop over one may have an empty iteration space, even with a negative size (e.g. in the overlap2 remainder regions outside of a SubDomainSet). _make_guard only checked the iteration space size for non-affine regions, so such loops were launched unguarded. With nvc, an OpenACC tile/collapse nest with one negative and one positive trip count can launch 2^31-1 empty gangs (~1.3 s each on an H200). Emit the existing size guard also for affine regions iterating over a SubDimension. Device loops over root Dimensions are unchanged. Fixes #3093
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3096 +/- ##
==========================================
- Coverage 84.40% 84.40% -0.01%
==========================================
Files 260 260
Lines 58153 58164 +11
Branches 4948 4948
==========================================
+ Hits 49086 49093 +7
- Misses 8245 8248 +3
- Partials 822 823 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3093.
A device parallel loop over a
SubDimensionorMultiSubDimensionhas bounds that depend on runtime thicknesses, so its iteration space can be empty, sometimes with a negative size. This happens, for example, in theoverlap2remainder regions outside aSubDomainSet._make_guardonly checked the size of non-affine regions, so these affine loops were launched unguarded. With nvc, an OpenACCtile/collapsenest with one negative and one positive trip count can launch 2^31-1 empty gangs. Each such launch takes about 1.3 s on an H200, which is whyoverlap2with aSubDomainSetran more than 100x slower thanbasic. #3093 has the details and a standalone C reproducer.Fix: emit the existing
symbolic_size > 0guard, a Cifaround the region, also for affine regions that iterate over aSubDimension. Device loops over rootDimensions, sparse loops, CPU code, and the CUDA/HIP backends are unchanged.For a
SubDomainSetloop, the generated code becomes:The guard is evaluated on the host, once per launch.
Tests:
TestMultiSubDomain::test_device_guard_empty_ispace[openacc|openmp]. It checks the guard in the generated code and runs without a GPU. It fails on main and passes here.test_composite_buffering_taskingandtest_composite_buffering_tasking_multi_outputcount iteration trees.op0now has one more, because the guardConditionalsits in the time loop.Evidence:
overlap2path, 1 rank, ~9 steps, Operator time:SubDomainSet:overlap2went from 23.22 s to 0.01 s;basicis 0.01 s.overlap2went from 58.07 s to 0.01 s;basicis 0.01 s.overlap2takes 3.78 s against 2.06 s forbasic; it was killed at 300 s before.basicand the whole-domain run.test_gpu_common.py,test_gpu_openacc.py,test_gpu_openmp.py,test_mpi.py,test_subdomains.py,test_operator.pyandtest_iet.pypass (CPU; the GPU modules skip).test_mpi.py,test_subdomains.pyandtest_operator.pypass serially under openmp.test_adjoint.py,test_gpu_openacc.py,test_gpu_common.py,test_subdomains.py::TestMultiSubDomainandtest_operator.py::TestEstimateMemoryunder OpenACC on the H200: main gives 289 passed and 2 failed, this branch 291 passed and 2 failed. The 2 extra passes are the new test, and the 2 failures are the same on both (TestMultiSubDomain::test_3D,test_subdomainset_mpi[(4, overlap)]).🤖 Generated with Claude Code