[ggorman]

Under DEVITO_MPI=overlap, overlap2 and full, an equation over a SubDomainSet (or DevitoPRO's ABox) skips points near the box edges. basic and diag2 are correct. This is the MultiSubDomain counterpart of #1309: #1320 fixed it for SubDimension only.

Root cause

HaloScheme.__init__ only honors d.is_Sub and not d.local (halo_scheme.py:145). MultiSubDimension is an AbstractSubDimension with is_Sub=False, so its thicknesses never reach honored or the omapper. The thickness is then added on top of the CORE/OWNED bounds instead of being intersected with them. Generated code on main for the new test (overlap2, 4 ranks):

const int x_ltkn0 = boxes[n0][0];
compute0(u_vec,r0,r1,t0,t1,x_M - 2,x_ltkn0,x_m + 2,x_rtkn0,...);    // CORE starts at x_m + 2 + x_ltkn0
...
for (int ix = x_m + x_ltkn0; ix <= x_M - x_rtkn0; ix += 1)         // inside compute0
...
compute0(u_vec,r0,r1,t0,t1,reg0[i].x_M,x_ltkn0,reg0[i].x_m,x_rtkn0,...);   // OWNED, same composition

overlap2/full have a second constraint. The OWNED region struct is filled in Python at argument time (MPIRegion._arg_values). For a SubDomain the struct also carries the per-region thickness, but a MultiSubDimension thickness is only bound inside the Operator (x_ltkn0 = boxes[n0][0], or per time step for an ABox).

Fix

• HaloScheme honors the thicknesses of every non-local AbstractSubDimension; AbstractSubDimension.local defaults to False.
• The overlap2 region describes only the OWNED split of the root Dimensions. The remainder intersects each region with the iteration bounds: ltkn -> MAX(0, x_m + ltkn - reg[i].x_m), and likewise on the right. This is the same rule for SubDimension and MultiSubDimension, with no separate runtime bookkeeping.
• MPIRegion loses its non-Dimension fields.

Generated code with the fix:

compute0(u_vec,r0,r1,t0,t1,x_M - 2,MAX(0, x_ltkn0 - 2),x_m + 2,MAX(0, x_rtkn0 - 2),...);
compute0(u_vec,r0,r1,t0,t1,reg0[i].x_M,MAX(0,x_m + x_ltkn0 - reg0[i].x_m),reg0[i].x_m,MAX(0,-x_M + x_rtkn0 + reg0[i].x_M),...);

For a regular SubDomain under overlap2/full, the region struct drops its *_tkn fields and the remainder computes the same intersection itself instead of reading it from the struct. basic, diag2 and overlap code for regular SubDomains is unchanged.

Test

TestMultiSubDomain::test_subdomainset_stencil_mpi: two boxes on a 12x12 grid over 4 ranks, one step of u.forward = u.laplace + 1. The expected result is exactly 1 inside the boxes and 0 elsewhere. It runs in all five modes. On main it fails under overlap, overlap2 and full; with the fix it passes in all five.

Checks (local, ese-ponte)

• tests/test_{mpi,subdomains,dimension,operator,buffering,gpu_common,interpolation,pickle}.py under DEVITO_LANGUAGE C, CXX and openmp, with the MPI tests run separately without xdist:

  	C	CXX	openmp
  MPI tests (-m parallel), main	592 passed, 1 xpassed; 3 failed (the new test under overlap/overlap2/full)	592 passed, 1 xpassed; 3 failed (same)	593 passed; 3 failed (same)
  MPI tests, this PR	595 passed, 1 xpassed	596 passed	596 passed
  other tests (-m "not parallel"), main and this PR	1333 passed	1333 passed	1333 passed

• ruff check --preview, isort and pre-commit run --files are clean.

• DevitoPRO with this commit pinned (devitopro#1026's ABox MPI tests included):

  • test_abox.py, test_subdomains.py and test_compressed_funcs.py::TestAdvanced::test_gradient_abox: 8 failed / 13 passed on the current pin (all ABox forward/backward/adjoint/gradient cases under overlap2/full); 21 passed with this commit. The non-MPI cases (37) pass on both.
  • GPU, test_gpu_lang.py::TestMPIOverlapMode (incl. test_alignment_core_and_remainder), H200: 39 passed and 2 failed with this commit, identical to the current pin. All 8 test_alignment_core_and_remainder cases pass on both; the 2 test_layered_domain_check[noop] failures are an nvcc build error (__cudaLaunch macro) on both.
  • recipes iso-acoustic gradient with -abox True, 4 ranks: |dvp| under full/overlap2 goes from 401.43 to 544.235, matching basic/diag2 and within 3e-7 relative of serial.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 11.42857% with 31 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.39%. Comparing base (6d2ad82) to head (ab62c88).
⚠️ Report is 18 commits behind head on main.

Files with missing lines	Patch %	Lines
devito/mpi/routines.py	0.00%	17 Missing ⚠️
tests/test_subdomains.py	12.50%	14 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3075      +/-   ##
==========================================
+ Coverage   84.28%   84.39%   +0.10%     
==========================================
  Files         258      260       +2     
  Lines       57469    58168     +699     
  Branches     4914     4947      +33     
==========================================
+ Hits        48439    49091     +652     
- Misses       8206     8256      +50     
+ Partials      824      821       -3     

Flag	Coverage Δ
pytest-gpu-aomp-amdgpuX	68.45% <10.52%> (-0.04%)	⬇️
pytest-gpu-gcc-	79.08% <11.42%> (+0.15%)	⬆️
pytest-gpu-icx-	79.00% <11.42%> (+0.14%)	⬆️
pytest-gpu-nvc-nvidiaX	69.15% <10.52%> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mloubout]

This doesn't look right, having to track MultiSubs seaprately looks overly complicated, and having a haloscheme having separate arguments and runtime_tkns is not right.

[ggorman]

The osx failure is the Apple-silicon detection flake, not this change: the job header reads platform: cpu64 on the arm64 runner, so the cpu64-arm skip does not apply and test_nested_nthreads fails compiling x86 intrinsic headers (https://github.com/devitocodes/devito/actions/runs/39303223750/job/118233531439). #3048 fixes the detection. The other 41 jobs pass.

[ggorman]

Reworked: there is no MultiSub tracking and no runtime_tkns any more. HaloScheme now honors every non-local AbstractSubDimension. Previously only is_Sub dimensions were honored, so MultiSubDimension was missed; that is the same gap #1320 closed for SubDimension.

The overlap2 region struct now holds only the OWNED regions of the root Dimensions. The remainder intersects it with the iteration bounds (MAX(0, x_m + ltkn - reg[i].x_m)), the same way for a SubDimension and a MultiSubDimension, so HaloScheme gains no new state.

The one codegen change for regular SubDomains: under overlap2/full the struct no longer stores per-region thicknesses, and the remainder computes them instead. The new test is a minimal two-box SubDomainSet stencil that fails on main under overlap/overlap2/full. The PR description has the generated code and the test matrix.