Repository navigation
GPU: send num_procs and proc_rank to the device after the rank is bound to its GPU - #2371
Draft
sbryngelson wants to merge 1 commit into
Draft
sbryngelson wants to merge 1 commit into
sbryngelson wants to merge 1 commit into
Conversation
|
Claude Code Review Head SHA: 63649fa Files changed:
Findings: |
Lines of Code
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contribution Policy
All contributions are expected to demonstrate:
If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.
Summary
In
simulation,s_mpi_initializeends with$:GPU_UPDATE(device='[num_procs, proc_rank]'), but it is called froms_initialize_mpi_domainbefore each rank is bound to its GPU (acc_set_device_num/omp_set_default_device). A rank bound to a device other than the default one never receives the values: its device copy ofnum_procsstays uninitialised. The only device reader,f_local_rank_owns_location(if (num_procs > 1)), then returns.true.on both ranks in the IBM collision kernels, so the contact force is applied by both ranks and summed.Fix: do the update right after the device binding in
s_initialize_mpi_domainand drop it froms_mpi_initialize.pre_processandpost_processalso calls_mpi_initialize, butGPU_UPDATEis a no-op there and nothing else in them reads the values on the device.Evidence (hand test, NVIDIA A100 80GB (Georgia Tech wingtip), NVHPC 25.5, OpenACC, 2 ranks)
-g 0 2maps rank 0 to GPU 0 and rank 1 to GPU 2 (-g 0puts both ranks on GPU 0). Quantities are density (var 1656,D/cons.1.00.000100.dat) and energy (var 220,D/cons.4.01.000100.dat) of a 2-rank IBM collision case (a friction test from #2354). Golden values (Cray CPU build): 0.973596 / 2.50569.-g00 200 22 00 30 1(two GPUs)0(one GPU)examples/1D_sodshocktube, 2 ranks, throwaway kernel printing device vs host at the first step (unpatched)num_procs/proc_rank= 0 / 0, host = 2 / 1; rank 0 fine--gpu mp, tests 09FDDC81 04FB9E91 9C0CAF25 50B98DBF (one GPU per rank, binding is a no-op)The patched mappings agree to about 1e-12. The unpatched
-g 0 2result is reproducible to 1e-12 across repeats;--rdma-mpidid not change it and it persists with friction off, so it is not a halo or friction issue. The unpatched build only passes when both ranks share a GPU.Not verified
gq3_pts/gq3_wtsandnvhpc_omp_state_dependent_eosmatched the host everywhere, sonum_procsandproc_rankare the only stale device values found.omp_set_default_device): checked by reading the code only, not run.--rdma-mpi(the latter made no difference to the unpatched failure on the A100).This PR was made with Claude Code (AI).