[sbryngelson]

Developed with AI assistance (Claude Code). Benchmarks and the regression were run on OLCF Frontier (MI250X, amdflang
OpenMP offload); commands are listed so each number can be reproduced.

Contribution Policy

All contributions are expected to demonstrate:

• A clear understanding of the codebase
• Alignment with product direction
• Thoughtful reasoning behind changes
• Evidence of real-world usage or hands-on experience with the problem

Summary

GPU-efficiency changes to the existing solver: same numerics, same results (every master golden passes unchanged), 18-31%
faster on the benchmark suite except IGR. This is the first of two PRs; the second (AMR) builds on it.

• Fewer per-launch mapping costs. On amdflang OpenMP, private work arrays were mapped tofrom on every launch. A new
  privateArrays option in the GPU macros maps them alloc (OpenACC merges them into private).
• Flat Riemann flux buffers. The Riemann solvers write module-level flat arrays that CBC, QBMM, surface tension,
  conduction and chemistry read directly, instead of scalar_field arguments; flux_n and flux_gsrc_n are gone.
• Pack once per RHS. The reconstruction input is packed once per RHS evaluation, not once per sweep direction.
• Restructured HLL/HLLC/LF/HLLD. HLLC face work runs in routines with explicit-shape arrays; WENO5 stencil loops have
  literal bounds so the stencils stay in registers.
• Copy primitives only for their readers. q_prim_vf is copied out of the RHS only when probes, IB, Lagrangian
  bubbles, synthetic turbulence or run-time info read it (hyper_cleaning's psi is always copied).
• EOS family is a build-time constant in every build. Stiffened-gas builds drop the state-dependent EOS chain from
  every kernel; a case whose EOS does not match its binary aborts with a message. CI builds a third simulation variant
  for the Mie-Gruneisen, JWL and Vinet cases. Case-optimized builds also compile only the selected Riemann solver.
• Tooling and small fixes found along the way are in Small fixes: bc_type dummy shape, serial BC files every save, format/lint/validator tooling #2118.

Verification

Regression

Frontier, amdflang OpenMP (./mfc.sh test -v --max-attempts 3 -a --gpu mp -- -c frontier_amd, 4 shards): 729/729,
no golden changed.

Pending here (CI covers these too): NVHPC 25.11 OpenACC suite and OpenMP build on A100; Frontier CCE OpenACC/OpenMP.

Benchmarks (./mfc.sh bench, one Frontier node, 8 GCDs, master and this PR alternated twice)

Grind time, lower is better (run-to-run spread under 1%):

Case	master	this PR	change
5eq_rk3_weno3_hllc	0.1917	0.1330	-30.6%
5eq_rk3_weno3_lf	0.1852	0.1321	-28.7%
ibm	0.4629	0.3416	-26.2%
5eq_rk3_weno3_hll	0.2003	0.1483	-26.0%
hypo_hll	0.1625	0.1249	-23.2%
viscous_weno5_sgb_acoustic	0.3775	0.3112	-17.6%
igr	0.3513	0.3736	+6.3%

IGR is slower: with the EOS family constant, the compiler inlines the EOS chain into the IGR Riemann kernel (76-80 VGPRs +
128 AGPRs instead of 24-28 + 128), one fewer wave per SIMD. A runtime flag restores IGR but gives back part of the gain
elsewhere (HLLC 0.133 -> 0.169, IBM 0.342 -> 0.370), so the constant is kept. The changes target AMD per-launch costs;
NVIDIA timings are not measured yet.

Commits

GPU macros; build-time EOS family; solver restructure; review fixes. Only the final tree is built and tested.

[github-actions]

Claude Code Review

Head SHA: e92747b

Files changed:

• 41
• .github/scripts/check_caseopt_shared_slugs.py
• .github/scripts/prebuild-case-optimization.sh
• .github/workflows/common/build.sh
• .github/workflows/test.yml
• cmake/Fypp.cmake
• cmake/GPU.cmake
• docs/documentation/contributing.md
• src/common/include/omp_macros.fpp
• src/common/include/parallel_macros.fpp
• src/common/m_eos.fpp

Findings:

• medium src/simulation/m_rhs.fpp: The new packed=id > 1 .and. .not. weno_Re_flux argument to s_reconstruct_cell_boundary_values makes s_weno/s_muscl skip re-packing v_rs_weno/v_rs_ws_muscl for sweeps 2 and 3. That is only correct if nothing writes that buffer between the id=1 and id>1 calls. The diff does not show this guarantee. Any other s_weno/s_muscl call in between, such as the iv%beg:iv%end branch, the viscous-derivative reconstructions, or s_reconstruct on other v_vf arrays, would leave MP-WENO/THINC/MUSCL reading stale data. The adjacent comment ("the viscous-derivative reconstruction reuses the buffer") also contradicts the guard, which excludes weno_Re_flux. Confidence: moderate, since I could not see the rest of the sweep loop. Suggested fix: confirm no intervening call overwrites the buffer. Otherwise drop the optimization or invalidate it explicitly. Also correct the comment.
• medium src/simulation/m_rhs.fpp: q_prim_vf is now copied out of q_prim_qp only for probe_wrt, ib, bubbles_lagrange, synthetic_turbulence and run_time_info on stage 1. The reader list is hand-maintained in a comment. Any other consumer of q_prim_vf after s_compute_rhs is not covered. Examples are cfl_adap_dt/cfl_const_dt in the dt computation, data output, and the m_time_steppers paths. Those consumers would see stale primitives with no error. Confidence: low to moderate, because m_time_steppers and m_start_up are not in the diff. Suggested fix: verify every post-RHS reader of q_prim_vf, or keep the previous unconditional copy for the CFL and save paths.
• medium src/common/include/shared_parallel_macros.fpp: JOIN_LISTS splices strings with a[:-1] + ', ' + b[1:]. It assumes both arguments are non-empty '[...]' literals with no stray whitespace or trailing text. An empty '[]' or a padded '[ ]' yields '[, x]' or '[ , x]', which is an invalid clause. A non-bracketed string is silently truncated. Suggested fix: #:assert the bracket form, or strip the brackets and rejoin the elements.

[github-actions]

Lines of Code

File	Lines	Diff
src/simulation/m_rhs.fpp	2034	+63
src/simulation/m_riemann_solver_lf.fpp	553	+32
src/simulation/m_riemann_solvers.fpp	141	+29
src/simulation/m_riemann_solver_hllc.fpp	1336	+28
src/common/include/omp_macros.fpp	391	+11
src/simulation/m_riemann_solver_hlld.fpp	199	+7
src/simulation/m_riemann_state.fpp	1182	-6
src/common/m_eos.fpp	520	-5
src/common/m_helper_basic.fpp	123	+5
src/simulation/m_weno.fpp	1331	+5
src/simulation/m_riemann_solver_hll.fpp	624	+3
src/simulation/m_cbc.fpp	1112	-2
src/simulation/m_muscl.fpp	189	+2
src/simulation/m_riemann_solver_hypo_hlld.fpp	785	+2
src/common/include/acc_macros.fpp	283	+1
src/common/include/parallel_macros.fpp	252	+1

Directory	Lines	Diff
common	10557	+13
simulation	28743	+163
total	47806	+176