Repository navigation
Block-structured adaptive mesh refinement (AMR) - #2117
Draft
sbryngelson wants to merge 12 commits into
Draft
sbryngelson wants to merge 12 commits into
sbryngelson wants to merge 12 commits into
Conversation
|
Claude Code Review Head SHA: f08f835 Files changed:
Findings:
Not reviewed in depth: the bulk of the |
sbryngelson
force-pushed
the
amr
branch
5 times, most recently
from
November 3, 2026 18:00
88c7622 to
fb72d12
Compare
…L/HLLC/LF/HLLD, copy primitives only for readers; case-optimized builds compile only the selected Riemann solver
…scalar reductions return their input; fail format on ffmt structure warnings; Cray debug bounds checking; Open MPI oversubscribe; stage-guard lint; validator fixes
Lines of Code
|
sbryngelson
force-pushed
the
amr
branch
2 times, most recently
from
November 4, 2026 04:36
be1f45c to
2dd3cca
Compare
…ry helpers, restart I/O
…M, IGR, physics, startup, output)
…iction/reflux, clustering, nesting, regrid, subcycling, restart)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2116 (solver performance) and #2118 (small fixes). Until they merge this diff includes their commits;
the AMR changes are the last 8 commits.
Contribution Policy
All contributions are expected to demonstrate:
Summary
Adds block-structured adaptive mesh refinement to
simulation, off by default (amr = F). With AMR off, results arethose of #2116 (every existing golden passes unchanged).
amr_max_grid_sizecells, refinement ratio 2 (or 4 for one level), up to
amr_max_levelproperly nested levels, regrid everyamr_regrid_intsteps.amr_subcycle, two fine steps per coarse step, up to two refinedlevels) with time-interpolated ghosts. Flux registers reflux every coarse/fine face, so the scheme is conservative.
amr_prolong = 2) with a fallback to linearwherever the quartic state would have non-positive density or internal energy; conservative restriction.
with
amr_place_owner). Exchanges are nonblocking waves with keyed tags, so results do not depend on message timing.Lagrangian bubbles (lock-step, cloud on the base grid), acoustic sources (base grid), 1D MHD, IGR (single rank), and
immersed boundaries (fixed, prescribed or force-driven motion).
docs/documentation/amr.md(user guide),amr_implementation.md(internals), one AMR section incase.md.The solver runs on a block by swapping its grid globals (
m,n,p, index ranges, coordinates) in and out;amr_implementation.mdstates the contract for module state that depends on the grid. AMR validation fails closed: withamr = Tthe validator rejects any logical switch not inAMR_VETTED_FLAGS, so new physics is refused until it meets thatcontract. Explicitly rejected with a message:
model_eqns = 1, Euler-Euler bubbles, QBMM, phase change, surface tension,synthetic turbulence, cylindrical or stretched grids,
bc = -4withnull_weights; subcycling with IGR, chemistry,Lagrangian bubbles or moving bodies.
Verification
Regression
-c frontier_amd, 4 shards)--no-mpi)-c frontier, 4 shards)-c frontier, 4 shards)The lanes ran on commits just before the head. Since then: an unused dummy argument removed, the no-MPI reduction
fallback from #2118 (not compiled in MPI builds), comments, two compiler-specific build flags and one test tolerance
(below). CI runs on the head. The 7 CCE misses are multi-rank AMR cases (np = 2 to 8) off by 1.5e-12 to 1.6e-9
against tolerances of 1e-12 to 1e-10: the goldens come from amdflang, and CCE's multi-rank sums differ in the last digits.
The same 7 cases miss by the same amounts before this PR's cleanup commits. Tolerances are unchanged.
New tests: 56 AMR cases (multi-rank, multi-level, subcycling, restart, IB, quartic prolongation, and AMR variants of the
chemistry, IGR and Lagrangian-bubble cases) and one
rank_time_wrtdiagnostics case. No existing golden changes.Compiler issues found and handled here:
m_amr_regridinto host loops it flattened and dropped stores in them(regrid read garbage block origins and faulted).
m_amr_regridjoins the existing per-file-Oipa0list inCMakeLists.txt.anyway; both are now automatic arrays.
-mp=gpu:fort2segfaults while writing the-Minforeport form_amr_exchange(reproduced in the CI container; the generated code is not involved). That file is built with
-Mnoinfounder NVHPCOpenMP.
-O3: the 1D acoustic-source case under dynamic regrid drifts 3e-12 from the golden, like its static-blockparent, and gets the same 1e-8 tolerance.
Accuracy and conservation
1D periodic Euler, WENO5, SSP-RK3, refinement ratio 2:
L1 density error after one period. Third order is kept; the error constant is larger because the coarse/fine ghost fill is
second order. Total mass, momentum and energy change by 1.4e-16, 4.2e-16 and 1.9e-16 (relative) over 200 steps with regrid
every 4 steps. The same run on 1 and 4 ranks is bitwise identical.
Performance (weak scaling, blast lattice, 8M base cells per GPU, one refined level, subcycled)
s/step (efficiency vs 8 GPUs):
MFC refines about 1.3x more volume than AMReX and PeleC here (40.5% vs 30.3%). The table is from an earlier commit of this
work; at 64 GPUs this PR's tree gives 0.686 and 0.687 s/step in two runs (table: 0.699), timed as
(t240 - t40)/200 of the step-loop wall so startup and the first regrids are excluded. Regrid cost
grows faster than the step beyond 512 GPUs; that is a follow-up.
Use
Mach 2.5 shock through 297 fixed random spheres: 384 x 256 x 256 base grid plus one level, 128 GPUs, 1350 steps, up to
about 5300 blocks.
Known limitations and follow-ups
migrated), which costs accuracy in multi-level runs that regrid often.
the swap but touches the whole solver.
m_amr_stateholds many module variables; a block-table derived type would be cleaner.