Skip to content

GPU: send num_procs and proc_rank to the device after the rank is bound to its GPU - #2371

Draft
sbryngelson wants to merge 1 commit into
MFlowCode:masterfrom
sbryngelson:fix-num-procs-device-binding
Draft

sbryngelson wants to merge 1 commit into
MFlowCode:masterfrom
sbryngelson:fix-num-procs-device-binding

Conversation

@sbryngelson

@sbryngelson sbryngelson commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Contribution Policy

All contributions are expected to demonstrate:

  • A clear understanding of the codebase
  • Alignment with product direction
  • Thoughtful reasoning behind changes
  • Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

Summary

In simulation, s_mpi_initialize ends with $:GPU_UPDATE(device='[num_procs, proc_rank]'), but it is called from s_initialize_mpi_domain before each rank is bound to its GPU (acc_set_device_num / omp_set_default_device). A rank bound to a device other than the default one never receives the values: its device copy of num_procs stays uninitialised. The only device reader, f_local_rank_owns_location (if (num_procs > 1)), then returns .true. on both ranks in the IBM collision kernels, so the contact force is applied by both ranks and summed.

Fix: do the update right after the device binding in s_initialize_mpi_domain and drop it from s_mpi_initialize. pre_process and post_process also call s_mpi_initialize, but GPU_UPDATE is a no-op there and nothing else in them reads the values on the device.

Evidence (hand test, NVIDIA A100 80GB (Georgia Tech wingtip), NVHPC 25.5, OpenACC, 2 ranks)

-g 0 2 maps rank 0 to GPU 0 and rank 1 to GPU 2 (-g 0 puts both ranks on GPU 0). Quantities are density (var 1656, D/cons.1.00.000100.dat) and energy (var 220, D/cons.4.01.000100.dat) of a 2-rank IBM collision case (a friction test from #2354). Golden values (Cray CPU build): 0.973596 / 2.50569.

Build -g density energy result
unpatched 0 0.973596 2.50569 pass
unpatched 0 2 0.973094 2.97088 FAIL
patched 0 0.973596 2.50569 pass
patched 0 2 0.973596 2.50569 pass
patched 2 0 0.973596 2.50569 pass
patched 0 3 0.973596 2.50569 pass
master (with #2353) + patch, 2-rank IBM tests 09FDDC81 04FB9E91 9C0CAF25 50B98DBF 5E746B78 0 1 (two GPUs) 5 passed, 0 failed
master (with #2353) + patch, same 5 tests 0 (one GPU) 5 passed, 0 failed
Direct measurement, NVHPC 25.11, examples/1D_sodshocktube, 2 ranks, throwaway kernel printing device vs host at the first step (unpatched) two GPUs rank 1 device num_procs/proc_rank = 0 / 0, host = 2 / 1; rank 0 fine confirms mechanism
Same measurement, both ranks on one GPU one GPU device = host as expected
Tuolumne (LLNL) MI300A, Cray CCE, --gpu mp, tests 09FDDC81 04FB9E91 9C0CAF25 50B98DBF (one GPU per rank, binding is a no-op) one GPU per rank 4 passed, 0 failed (unchanged)

The patched mappings agree to about 1e-12. The unpatched -g 0 2 result is reproducible to 1e-12 across repeats; --rdma-mpi did not change it and it persists with friction off, so it is not a halo or friction issue. The unpatched build only passes when both ranks share a GPU.

Not verified

  • No CI test covers this: CI GPU runners have few GPUs, so 2-rank tests run both ranks on the same GPU (or one per GPU on the default device), which hides the bug.
  • No H100/H200 two-GPU runs.
  • In the direct measurement, gq3_pts/gq3_wts and nvhpc_omp_state_dependent_eos matched the host everywhere, so num_procs and proc_rank are the only stale device values found.
  • The OpenMP-offload path (omp_set_default_device): checked by reading the code only, not run.
  • AMD GPUs and --rdma-mpi (the latter made no difference to the unpatched failure on the A100).
  • A regression test: it would need 2 ranks on 2 different GPUs, which CI cannot provide.

This PR was made with Claude Code (AI).

@github-actions

Copy link
Copy Markdown

Claude Code Review

Head SHA: 63649fa

Files changed:

  • 2
  • src/common/m_mpi_common.fpp
  • src/simulation/m_start_up.fpp

Findings:
No issues found. Checked src/common/m_mpi_common.fpp and src/simulation/m_start_up.fpp (the moved GPU_UPDATE now follows device binding; num_procs/proc_rank are declared on device only in simulation's m_global_parameters).

@github-actions

Copy link
Copy Markdown

Lines of Code

File Lines Diff
src/common/m_mpi_common.fpp 1578 -1
src/simulation/m_start_up.fpp 1535 +1
Directory Lines Diff
common 12518 -1
simulation 41026 +1
total 61096 +0

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant