diff --git a/knowledge/KNOWLEDGE.md b/knowledge/KNOWLEDGE.md new file mode 100644 index 0000000..14f34da --- /dev/null +++ b/knowledge/KNOWLEDGE.md @@ -0,0 +1,769 @@ +# KNOWLEDGE.md — MOM6 GPU Porting: Agent Startup Knowledge Base + +> **Who this is for.** An agent (or human) booting into the `dev/gpu` fork of MOM6 with zero prior +> context, tasked with continuing — or finishing — the GPU port. This file is the operational +> playbook: identity and ground rules, a compact architecture orientation, the end-to-end porting +> procedure, decision rules, a symptom→fix debugging index, the prioritized work queue, the +> proven-works and never-do inventories, the verified findings, and the genuinely open questions. +> Depth lives in `docs/gpu-knowledge/00-…14-*.md` (index in §10); every load-bearing rule here is +> inlined with its citation so you can act without opening the deep docs, and every section links +> to them for the full evidence. +> +> **Repo state.** Branch `dev/gpu`; upstream baseline `dev-gfdl`. `git diff dev-gfdl...dev/gpu` = +> the totality of merged GPU work (320 commits, 52 files, +8310/−4615). Side branches +> (`kblock-hor-visc`, `bodner-naive-port`, `port/pressureforce-benchmark_ALE`, +> `diag_map_mediator_port`, `remotes/edoyango/*`, `remotes/origin/*`) carry the in-flight work. +> All file:line references are against `dev/gpu` HEAD unless a commit hash or branch is named. + +--- + +## 1. Identity and non-negotiable ground rules + +This is the **dev/gpu fork of MOM6** (Modular Ocean Model 6), being ported to **NVIDIA GPUs** with +**NVHPC nvfortran**, using **OpenMP target offload** for data management and **Fortran +`do concurrent`** for compute. The rules below are not preferences; every one has been enforced in +review and has commits behind it. + +1. **Bitwise reproducibility is mandatory.** No refactor may change the order of floating-point + operations. Two runs (CPU vs GPU, pre- vs post-refactor, 1 vs N ranks) must produce + bit-identical fields, verified by `MOM_checksums` popcnt checksums (`MOM_checksums.F90:2680`, + `bc_modulus=1e9` `:112`) and EFP reproducing-sum energy output (`MOM_coms.F90`). A single + differing bit means the port is wrong — never accept a nonzero diff as "rounding" + (doc 05 §7.2, doc 07 §5–6). + - The escape hatch for restructuring: **extract arithmetic into `pure`/`elemental` + procedures verbatim** — moving code, never reordering it (doc 08 §4; `efp_decompose` + `MOM_coms.F90:778`, EOS `_loc` kernels). + - Producer loops and the reductions that consume their outputs must stay **fused**: splitting + them lets a compiler re-associate the accumulation (ifort did, even at `-O0` — commit + `5f413739b`, doc 09 §3). +2. **Preserve CPU performance.** One source form must serve both targets. The accepted mechanism is + **k-blocking / i-j tiling** with runtime block-size CS parameters whose defaults diverge at + compile time: `#ifdef __NVCOMPILER_OPENMP_GPU` → `0` (whole extent) on GPU, real cache-tile + sizes on CPU (`32/4/1` for continuity, `nkblock=1` for CorAdCalc/hor_visc) + (`MOM_continuity_PPM.F90:3120-3129`; doc 05 §2). `0` is resolved to the full extent at the point + of use, never stored back into the CS. +3. **`do concurrent` is the default parallel idiom.** OpenMP `target teams` compute constructs are + used only for (a) serial-in-k column recurrences that need `collapse(2)` + `private` scratch, + and (b) regions where nvfortran demonstrably mis-schedules DC (manual `num_teams`, commit + `5b5f6b2b1`). The traffic has run both directions (`e8b0ecfbf` "omp target teams loop -> do + concurrent") — escalate only on evidence (doc 04 §3, §5.5). +4. **No runtime polymorphism on device.** `class(...)`/type-bound dispatch inside a + `do concurrent`/`target` region causes runtime errors or mishandled implicit copies of `this` + on nvfortran. The EOS layer is the case study and the `_loc` free-function duplication is the + fix (doc 06). ALE's `Recon1d` class dispatch is sidestepped via the non-polymorphic OM4 + select-case path (doc 14 §7). +5. **Expect real compiler bugs.** NVHPC 25.5 (A100 crash on conditional map of an optional dummy, + `2108e0eba`), 25.9 (silently wrong `do concurrent` result, `MOM_tracer_hor_diff.F90:1464`), and + 25.11 (early-`exit` miscompile, `e23d6a7b1`) have all produced genuine, version-identifiable + defects. Document every workaround with a comment and a commit; but **check your own diff for a + missing map or misplaced accumulation before blaming nvfortran** (doc 13 row 24). +6. **Constraints for research/porting agents:** study source + git only unless explicitly told to + build; never write outside the repo (temp files → `tmp_local_artifacts/` at repo root). + +--- + +## 2. Architecture orientation (compact) + +Full treatment: `docs/gpu-knowledge/00-architecture.md` (layout, call tree, inventory) and +`01-memory-control-structures.md` (memory model, CS graph). + +### 2.1 Layout + +`src/core/` — dynamical core (`MOM.F90` driver, `MOM_dynamics_split_RK2.F90`, +`MOM_continuity_PPM.F90`, `MOM_CoriolisAdv.F90`, `MOM_PressureForce_FV.F90`, +`MOM_barotropic.F90`, `MOM_variables.F90`, grids). `src/parameterizations/{lateral,vertical}/` — +physics (hor_visc, thickness_diffuse, MLE; vert_friction, set_viscosity, diabatic stack). +`src/ALE/` — vertical Lagrangian remap/regrid. `src/equation_of_state/` — EOS. +`src/tracer/` — registry + advection/diffusion. `src/framework/` — domains, comms +(`MOM_coms.F90` EFP sums), diag mediator, restart, checksums, `do_concurrent_compat.h`, +`MOM_memory_macros.h`. `config_src/infra/FMS2/` — FMS shims (incl. the GPU-aware +`do_group_pass`). `config_src/memory/` — `SYMMETRIC_MEMORY_` on/off (one `#define`). + +### 2.2 The time-stepping hot path + +`step_MOM` (`MOM.F90:522`) → `step_MOM_dynamics` → **`step_MOM_dyn_split_RK2`** +(`MOM_dynamics_split_RK2.F90:302`, predictor–corrector): +PressureForce (`:527/:909`) → CorAdCalc (`:589/:972`) → vertvisc_coef/remnant → btcalc → +continuity (`:695/:853/:1148`) → **btstep** (`:726/:1023`, inner barotropic sub-cycle, +`MOM_barotropic.F90:480`, wide-halo `CS%BT_Domain` "march inward", doc 09 §1.2) → +hor_visc (`:962`) → vertvisc — with ~10 grouped halo passes, all +`do_group_pass(..., omp_offload=.true.)`. Then `step_MOM_thermo` → `diabatic` (**host-only**) and +ALE remap (**host-only**, bracketed `update from(u,v,h)`/`to(u,v,h)` at `MOM.F90:1036/1038`), then +`step_MOM_tracer_dyn` → `advect_tracer`/`tracer_hordiff` (ported). The pure compute kernels contain +no halo updates and no reproducing sums by design — communication is hoisted into the drivers. + +### 2.3 The control-structure (CS) pattern and memory + +- Every module owns a `_CS` derived type holding all persistent state; populated by + `_init`, torn down by `_end`. Root: `MOM_control_struct` (`MOM.F90:204`, a **plain + value** owned by the driver, `map(alloc: MOM_CSp)` in `MOM_driver.F90:282` *before* + `initialize_MOM`). Dycore hub: `dyn_split_RK2_CSp` (pointer, allocated+mapped at + `MOM.F90:3258-3259`), which embeds `hor_visc`/`continuity_CSp`/`CoriolisAdv`/`barotropic_CSp` by + value and points to `vertvisc_CSp`/`set_visc_CSp` (doc 01 §3, §5 for the full tree with line + numbers). +- Array members are **macro-allocatable** (`ALLOCABLE_`, `ALLOC_(x)` → `allocate(x)`; + `MOM_memory_macros.h`). `pointer` members exist only for restart-registry targets and + cross-module aliasing (`MOM_variables.F90:294` comment) — see doc 02 for the aliasing hazard map. +- Index conventions: computational `isc:iec/jsc:jec`, data (=comp+halo, `NIHALO_=2`) + `isd:ied/jsd:jed`; B-grid (velocity/corner) `IsdB:IedB` with `IsdB = isd-1` in symmetric mode + (`MOM_hor_index.F90:89-99`). Shapes: `h(isd:ied,jsd:jed)`, `u(IsdB:IedB,jsd:jed)`, + `v(isd:ied,JsdB:JedB)`, `q(IsdB:IedB,JsdB:JedB)`. Dynamic-mode symmetric offsets live in the + runtime `ALLOC_` bounds, not the type declaration — always take bounds from + `G`/`HI` (`IsdB`, never hand-rolled `isd-1`) (doc 01 §1). +- Directive totals on `dev/gpu` `src/`: 698 `do concurrent`, 213 `target enter data`, + 168 `exit data`, 398 `target update`, 21 `declare target` (docs 03, 04). + +### 2.4 What is ported, in flight, and untouched (quantified) + +- **Ported (merged):** continuity (fully k/i/j-blocked), barotropic (most directive-dense, 242 DC), + CoriolisAdv (k-blocked, `b8c471cfa`), vert_friction + set_viscosity (teams-loop tridiagonals), + hor_visc (ported; full k-block on branch), tracer advection + hor_diff (incl. multi-GPU fixes), + EOS Wright(buggy) + Roquet_rho 2D/3D direct paths, reproducing sums (`8593a732a`), + PressureForce_FV, `find_uv_at_h` (`MOM_diabatic_aux.F90` — the diabatic-stack template), + `cuberoot`/`nth_root` intrinsics, `find_eta`. +- **In flight (branches):** `kblock-hor-visc`, `port/pressureforce-benchmark_ALE` (PLM density + integrals), `bodner-naive-port` (naive contrast case), `diag_map_mediator_port`, + `edoyango/port-set_diffusivity` (pre-device groundwork), `edoyango/port/thickness_diffuse` + (real DC directives, 17-file footprint), `origin/epbl-3d` (naive whole-column, "100x"), + `origin/jorge/diagnostics_port` (ALE OM4-chain `declare target` + `NK_GPU_MAX=500`), + `edoyango/acc-btstep` (OpenACC async experiment), `fix/nan_repro_sum`. +- **Untouched (0-diff vs dev-gfdl):** `MOM_set_diffusivity.F90`, `MOM_CVMix_KPP.F90`, + `MOM_energetic_PBL.F90`, `MOM_mixed_layer_restrat.F90`, `MOM_regridding.F90`, + `MOM_remapping.F90`, `MOM_diag_mediator.F90`, `MOM_restart.F90` — the whole diabatic stack, + ALE remap machinery, and diagnostics/restart IO (docs 12, 14). + +--- + +## 3. THE PORTING PROCEDURE — end-to-end recipe for one module + +Follow the numbered steps in order. Each inlines its load-bearing rules and cites the deep doc for +the full pattern. This is the distillation of every merged port. + +### Step 1 — Pick the target from the work queue (§6) + +Prefer Tier-1 items with an existing precedent (e.g. a diabatic tridiagonal → copy `find_uv_at_h`). +Check the in-flight branch table first (§2.4, doc 14 §9): if a branch already has groundwork +(j-blocking, `declare target` prep), build on it rather than restarting. + +### Step 2 — Pre-flight hazard audit (read the module before writing anything) + +Grep the module and record: +- **Pointer members and `associated()` control flow** (doc 02): which CS/type members are + `pointer` (restart targets? cross-module aliases like `tv%T => CS%T`?). Every pointer read on + device needs `map(to:)` (never `alloc`) and an `if (associated(...))` guard; every allocatable + needs `if (allocated(...))` — the intrinsics are not interchangeable (doc 02 §3). +- **Restart-registered fields** the module mutates: they need a `target update from` dominating + every `save_restart` site (doc 12 §8.2). +- **EOS calls**: which `form_of_EOS` paths are exercised? Only buggy-Wright and Roquet_rho have + GPU-safe direct 2D/3D kernels; the default `Wright_full` and 6 others are still polymorphic and + device-fatal (doc 06 §3). If the module needs EOS on device for another form, port that form + first with doc 06 §6.3's 6-step `_loc` recipe. +- **Recurrences**: identify every loop where `x(k)` depends on `x(k±1)` (tridiagonal solves, + cumulative sums, early-exit searches) — these dictate the loop form (§4.1) and disqualify naive + k-in-header parallelization (doc 05 §7.0 disqualifier list). +- **Halo calls**: `pass_var`/`pass_vector` (host-only, no `omp_offload` parameter exists) vs + grouped passes — plan the promotion (Step 7). +- **Diagnostics** (`post_data`) and debug checksums: every one is a device→host transfer to plan + (Step 8). +- **Float reductions**: any `+` over reals that parallelization would reorder → must stay serial, + or route through the EFP reproducing sum (doc 07 §6.1). If a block boundary would fall inside a + float sum, **stop and redesign**. + +### Step 3 — Apply the blessed k-blocking template (doc 05 §7, 11 steps, summarized) + +Qualify first (doc 05 §7.0): outer `do k=1,nz` with per-layer 2-D work and **no cross-layer +coupling**; vertical reductions either associative (max/or/int) or kept as an unsplit serial +`do k=1,nz` float sum. Then: +1. Add `integer :: nkblock` (and/or `niblock/njblock`) to the CS + (`MOM_continuity_PPM.F90:76-78`). +2. `#ifdef __NVCOMPILER_OPENMP_GPU` defaults: `0` GPU / `1` (or `32/4`) CPU (`:3120-3129`). +3. `get_param("MYMOD_NKBLOCK", ..., layoutParam=.true.)`; negative → `MOM_error(FATAL,...)`. +4. Resolve `0`→whole extent **at point of use**: `nkblock = merge(GV%ke, CS%nkblock, + CS%nkblock==0)` (`MOM_CoriolisAdv.F90:275`); expose an accessor if callers are external + (`hor_visc_nkblock`, `d6fe494e6`). +5. Shrink full-column scratch to block extent: `dimension(...,max(1,nkblock))` + (`MOM_continuity_PPM.F90:2706`) — never let a `0` sentinel reach an array bound. Beware: with + many large stack arrays, **declaration order itself moves CPU perf** ("Move with caution!" + comment; `28eb296f4`, doc 05 §4). +6. Outer host loop over block starts: `do k_start=1,nz,nkblock ; k_end=min(...) ; + kmax=k_end-k_start+1`. +7. Device kernel with a block-local index; either convention is bitwise-equivalent: + iterate `kk=1:kmax` with `k = k_start+kk-1` under `DO_LOCALITY(local(k))` + (`MOM_CoriolisAdv.F90:383`), or iterate global `k` computing `kk` (continuity, hor_visc). + **Index-mapping rule: scratch arrays use `kk`; full-size in/out and grid metrics use global + `k`.** Whichever index is not the DC control variable must be `local(...)` — otherwise it is a + shared-write race. +8. Thread the active sub-range into every helper (`PPM_limit_pos(...,ks,ke)` — a helper still + iterating `1:nz` against block-sized scratch reads out of bounds). +9. Incremental porting is allowed: unported sub-blocks stay as serial + `do k=k_start,k_end ! TODO: port` loops *inside* the installed block nest (CorAdCalc OBC/WENO, + hor_visc Leithy) — the block structure and the body port are separable steps. +10. Data mapping per Step 5. +11. Manual `num_teams` only on evidence (§4.5). + +Bitwise argument to re-verify for your case: blocking changes only which loop an operation nests in +and which buffer holds an intermediate; the per-point expression and every stencil read are copied +verbatim; `kk` is a bijection of `k` — so no FP op is reordered (doc 05 §3). Loop *fusion* is safe +only when the fused loops share an identical iteration space, write distinct arrays, and any +intra-set read is same-index within the same iteration (doc 05 §4, the `28eb296f4` fusion). + +### Step 4 — Device-callable helper rules (doc 08 §7 checklist) + +For every procedure called from inside a device region: +1. **Side-effect-free?** No module-state writes, no I/O. If not: extract the arithmetic into a new + `pure`/`elemental` procedure returning error flags via `intent(out)` args (the `efp_decompose` + model — `carry_overflow` couldn't be `pure` because it writes `overflow_error`; + `fix/nan_repro_sum` commit `939d06704` "cant be pure" relearned this). +2. **No polymorphic dummy?** A `class(...) :: this` actual in a device loop → duplicate as a free + `_loc` function with `this` deleted and the body **copied verbatim** (`52a1b3954`; + doc 06 §6.3). +3. **Which idiom is the call inside?** Bare `do concurrent`: nvfortran's lowering generally handles + visible `pure`/`elemental` calls. `!$omp target teams`/`loop`: the callee **must** be + `!$omp declare target` or force-inlined — getting this wrong is **silently wrong numbers, not a + build error** (`3cb184edd`: "inlining of ratio_max and flux_elem is MANDATORY … Otherwise + results are incorrect"). +4. **Directive state at HEAD:** force-inline is `!DIR$ ATTRIBUTES FORCEINLINE :: ` + (`MOM_continuity_PPM.F90:1086,1149`; `93dbbd36e`). There are **zero** `!NVF$ INLINE` directives + left in the tree (that form and the `-Minline=name:` flag list are historical). `ratio_max` + currently has *no* directive — a deliberate removal that rests on implicit device codegen + (§8, "`ratio_max`'s missing directive"); add `FORCEINLINE` for parity, and do not imitate the + gap in new code. +5. `declare target` placement: after all declarations, before the first executable + (`MOM_set_viscosity.F90` even repeats it harmlessly). Same-module vs cross-module is irrelevant + — the boundary is the lexical scope of the `target` construct. +6. **Verify** (Step 9) — there is no compile-time signal for a helper that silently failed to + inline. + +### Step 5 — Data-mapping lifecycle (doc 03) + +There is **no central mapping utility** — every directive is hand-written next to its +`ALLOC_`/`DEALLOC_` (deliberate: kind varies per array, balance must be visually auditable; +doc 03 §4). The canonical lifecycle: +```fortran +ALLOC_(CS%x(IsdB:IedB,jsd:jed,nz)) ; CS%x(:,:,:) = 0.0 +!$omp target enter data map(to: CS%x) ! in *_init (map(alloc:) if device writes first) +... +DEALLOC_(CS%x) +!$omp target exit data map(delete: CS%x) ! in *_end, mirrored member-by-member +``` +(`MOM_dynamics_split_RK2.F90:1350-1368` / `:2065-2083`.) +- **Shell before members, teardown in reverse.** Map the bare CS shell `map(alloc: CS%child)` + *before* calling `child_init` (pointer children need host `allocate` first); nested sub-types + one level at a time (`CS%pbv` then `CS%pbv%por_face_areaU`, `MOM.F90:3225-3231`). Scalar CS + members need no separate map — they ride in the shell and are refreshed with whole-struct + `!$omp target update to(CS)` after batches of host scalar writes (diag IDs, `dtbt`; + `MOM_barotropic.F90:6576`). +- **Map the parent exactly once; never re-`enter data` a parent after members are attached** — a + second whole-struct `map(to:)` clobbers member attachments and `associated()` state + (`c82e1254a`: the `visc` "allocated twice" bug was a double *device mapping* at + `MOM.F90:3278`+`:3709`, not a host double-allocate; doc 02 §5.2). Correct order: allocate host → + `map(alloc:)` parent once → register restarts (binds pointers) → `update to(parent)` → + `map(to: member) if (associated(member))`. +- **`map(to:)` vs `map(alloc:)`:** any struct or array whose host-set contents (including pointer + descriptors read by `associated()`) are read on device must be `to`; `alloc` is only for pure + workspace written on device before any read. `map(alloc: Reg, Reg%Tr(:))` was the multi-GPU + answer-change bug (`a774eb331`; doc 02 §4b). +- **Scratch:** subroutine-scoped `enter data map(alloc:...)` at entry, `exit data + map(delete/release:...)` at return; early-delete once last use passes is fine + (`up,vp` at `:1253`). Any array touched inside a device region needs an explicit map *before* + first touch — omission compiles fine and silently reintroduces per-statement traffic + (`bc05a6a89`). +- **Balance discipline:** every edit to an enter-data list must grep the same routine for the + paired exit-data list (`15ca2a25f` leaked `b_denom_1`). Neither `delete` nor `release` copies + back — host-needed values require `update from`/`map(from:)` first. +- **`delete` vs `release` (load-bearing):** `delete` forces the refcount to zero — a per-call + `delete` inside a callee destroys any outer persistent mapping of the same object (live example: + `vertvisc`'s delete kills `initialize_MOM`'s ADp map; §8, "The `ADp` mapping lifecycle"). Use `release` for + scoped/per-call teardown; `delete` only in the owning `*_end`. And **a `map(to:)` on an + already-present object does not refresh device contents** — refresh is `update to(...)` only + (§8, "A `map(to:)` on an already-present object does not refresh device contents"). +- **Flatten arrays-of-structs:** never map a derived-type array whose elements each hold an array, + inside a loop — one attach/detach per element (`1865612de` halved GPU time by flattening + `type(p2d), dimension(SZJ_)` to a flat 3-D array, +20% memory). The tolerated exception is the + tracer registry `Reg%Tr(:)` per-tracer mapped loop (`MOM_tracer_hor_diff.F90:209-216`). +- **Don't pass a CS by `pointer` into leaf routines** — plain derived-type dummies avoid + per-call descriptor "microtransfers" (`d75e4870e`). + +### Step 6 — Loop-form selection + +Apply the decision tree in §4.1 to every loop in the module. Reductions: scalar target only +(nvfortran can't reduce array elements — `MOM_tracer_hor_diff.F90:962`); commit to CS/module state +once, after the loop (`CS%ntrunc = CS%ntrunc + ntrunc`, doc 04 §4.2). No early `exit`/`return` +inside device-offloaded loop bodies — rewrite as an `if`-guard (`e23d6a7b1`, NVHPC 25.11 wrong +answers). + +### Step 7 — Halo strategy (doc 11 §9) + +- A module calling `pass_var`/`pass_vector` (no `omp_offload` parameter exists on those entry + points) has two options: + **A (cheap, cold paths):** keep the pass, bracket it `update from(field)` / `update to(field)`. + **B (blessed, hot paths):** add a `type(group_pass_type)` CS member, register fields with + `create_group_pass(CS%pass_x, field, G%Domain, halo=)` (batch several fields on one + handle = one MPI message), replace with `do_group_pass(CS%pass_x, G%Domain, omp_offload=.true.)`. + Preconditions: fields already `enter data`-mapped; FMS2 build (the flag forwards to the external + FMS `mpp_do_group_update` — §8, "FMS `omp_offload` is a genuine device path"). +- **Size `halo=` to the consuming stencil**, not reflexively `NIHALO_=2` (`pass_eta` uses + `halo=1`); if your port widens a stencil, widen the pass (doc 11 §9.2). +- The nonblocking path (`start_group_pass`/`complete_group_pass`) has **no** offload awareness: + it is the mutually-exclusive `else` branch and must be hand-staged. 14 of the 26 + `omp_offload=.true.` sites are gated behind `if (G%nonblocking_updates)`; 12 (incl. the + barotropic inner sub-cycle `:2757` and all tracer passes) are unconditional (doc 11 §4). +- For a solver with many cheap sub-steps, consider the wide-halo clone + march-inward pattern + (`clone_MOM_domain(..., min_halo=wd_halos)`, `MOM_barotropic.F90:6104`; `O(nstep)` → + `O(nstep/num_cycles)` exchanges) — but wide-halo residency is all-or-nothing: one host-resident + wide-bound array defeats the whole optimization (doc 09 §6.5.4). + +### Step 8 — Diagnostics / transfer discipline (doc 12 §8) + +- `post_data` and the whole diag mediator are **host-only** (zero directives, unchanged on + dev/gpu). The transfer is the producer's job. +- **Decouple transfer from post:** one `!$omp target update from()` covering all consumers, + guarded `if (CS%debug .or. CS%id_a>0 .or. CS%id_b>0)` (or `any([...] > 0)`), then the individual + `if (id>0) call post_data(...)` guards (`MOM_tracer_hor_diff.F90:719-733`, + `MOM_diagnostics.F90:1825-1827`). +- At coarse sync points the codebase default is a **blanket** transfer + (`MOM.F90:1091 update from(u,v,h,CS%uhtr,CS%vhtr)` before `calculate_diagnostic_fields`); match + it unless profiling shows a stall — the guarded push-down (`feat/new-diag-manager` `b2a30750a`) + is the direction of travel, not the current norm. +- Bracket unavoidable host-only detours: `from(...)` before when the host reads device data, + `to(...)` after when the host modified data the device needs (ALE `MOM.F90:1036/1038`; + `write_energy` `to(tv%S,tv%T)` `MOM_sum_output.F90:762`). +- **Every debug checksum on a mapped array needs its own `update from` immediately upstream** — a + transfer for a *different* array does not cover it (`b29b27150`; doc 07 §5.2). Restart-registered + device-mutated arrays need a dominating transfer before `save_restart` (`MOM_restart.F90` does + none of its own; doc 12 §6, and §8, "Restart staleness is latent"). +- The derived-type deep-copy trap: touching `CS%tv%T` through the container in an offload + pass materializes many small implicit transfers — wrap in an explicit + `map(to: CS%tv, CS%tv%T, CS%tv%S)` bracket or alias to a bare pointer first (`ff86497d5`). + +### Step 9 — Bitwise verification (mandatory acceptance gate; doc 07 §6, doc 05 §7.2) + +1. Build pre- and post-change at the same optimization level; run and compare `MOM_checksums` + field checksums (`hchksum`/`uchksum`/`vchksum`) at matching steps — **every checksum + bit-identical**, plus the EFP `write_energy` output. +2. Confirm block-size invariance: `nkblock=0`, `=1`, `=nz` must all agree. +3. Validate at **≥2 ranks/GPUs** and compare across GPU counts — single-GPU correctness does not + prove a port (missing `reduce` and `alloc`-vs-`to` bugs are latent until multi-device; + doc 11 §9.5). +4. On a checksum "mismatch", check the missing-`update from` stale-host case *first* (doc 07 §6.4), + then your own diff (map balance, misplaced accumulation), then §5's symptom index. +5. Strip debug prints; run the naive-vs-blessed review rubric (doc 10, 8 gates) against your diff + before proposing merge. + +--- + +## 4. DECISION RULES + +### 4.1 Loop-form decision tree (doc 04 §5.5 — take the first matching branch) + +1. **Pure elementwise, no loop-carried scalar, nothing aggregated** → bare `do concurrent`, + `k`/`kk` folded into the header. (~86% of all DC loops; `MOM_continuity_PPM.F90:430`.) Don't + add clauses you don't need. +2. **Per-iteration scalar temporaries (written before read)** → `DO_LOCALITY(local(...))` + (`MOM_CoriolisAdv.F90:383`, the k-blocking `local(k)` idiom). +3. **Private scalar must start with its pre-loop value** (conditionally overwritten, read + unconditionally) → `DO_LOCALITY(local_init(...))` (only 2 uses tree-wide; + `MOM_set_viscosity.F90:681-682`). +4. **Aggregate across the iteration space** → + - scalar (or whole-array) target: `DO_LOCALITY(reduce(: var))`; commit to persistent state + once after the loop. + - indexed array element target: **rejected by nvfortran** — stage a scalar + (`local(itmp)` outer + `reduce` inner + `a(j)=itmp`; `MOM_tracer_hor_diff.F90:959-966`). + - float `+` over reals: **not safe at all** — keep serial or use EFP (§4.4). +5. **Genuine serial-in-k recurrence** (tridiagonal, cumulative) → + `!$omp target teams loop collapse(2) private()` over `(I,j)` with a plain + serial `do k` inside (`MOM_vert_friction.F90:737-786`; `find_uv_at_h`). Right-way nesting: + `do concurrent(j) → serial do k → do concurrent(i)` — never a serial `do k` wrapping a full 2-D + DC (the `2a99c9dd1` fix). A data-dependent early-`exit` column search (not a recurrence) may + stay a serial `do k` inside a DC (`MOM_set_viscosity.F90:697`). +6. **Measured GPU under-subscription** on a hand-written teams region → pin + `num_teams(ceiling(real(tile_iters)/128.))` (`5b5f6b2b1`, 17→~238 teams; + `MOM_continuity_PPM.F90:701,707`), optionally `thread_limit(128)` + (`MOM_set_viscosity.F90:803`). Surgical, evidence-only; revert teams→DC where DC schedules fine. + +### 4.2 Map-kind selection (doc 03 §3.5) + +| Kind | Use when | +|---|---| +| `map(to:)` | Host-initialized data read on device before first device write; **always** for structs whose scalars/pointer descriptors feed device control flow | +| `map(alloc:)` | Pure workspace, first access is a device write; CS shells | +| `map(from:)` | One-shot device→host copy-out before `post_data`/host math | +| `map(delete:)` | Mirrored CS-member teardown in `*_end` (forces refcount to 0; no copy-back) | +| `map(release:)` | Scratch teardown where conditional branches make the map count uncertain (decrement; no copy-back) | +| `update to/from` | Refresh an existing mapping across a host-only detour; `update to(CS)` for batch scalar refresh | + +Guard intrinsics: `if (associated(x))` for pointers, `if (allocated(x))` for allocatables — never +mixed. Never `map(...) if (present(optional_arg))` inside a callee (A100/25.5 crash, +`2108e0eba`) — map optional buffers at the call site where presence is unambiguous. + +### 4.3 Pointer vs allocatable (docs 01 §7.1, 02 §7) + +- New persistent arrays: **macro-allocatable**, never pointer. Pointer only for restart-registry + targets or genuine cross-module aliasing. +- New child CS: `allocatable` unless trivially small and always-present (embedded value only for + the zero-array `continuity_PPM_CS` shape). Conditional allocation, independent device lifetime, + or `associated()`-gated flow control → `allocatable` (the `81680c15d`/`c82e1254a` rule). +- Local shorthand aliases (`eta => CS%eta`) are removable noise; *reassigned* aliases + (`p_surf`, `u_ptr` feeding btstep) are load-bearing and keep their per-call guarded maps. + +### 4.4 Reduction-operator safety (doc 07 §6.1) + +`+` over reals in a parallel loop → **never** (order-dependent). `max/min/.and./.or.` over anything, +and `+` over exact `int64` within a carry-safe block → safe. Reproducible real sums → only via +`reproducing_sum`/`reproducing_sum_EFP` (`MOM_coms.F90:80-90`); keep multi-call totals in +`EFP_type` until the last conversion. Never add a size-dependent host fallback branch to a device +reduction (the `fix/nan_repro_sum` NaN). + +### 4.5 Guarded vs blanket transfers (doc 12 §8.1) + +Blanket `update from` at sync points is the codebase default for wide fan-outs; guarded +(`if (id>0 .or. debug)`) per-field transfers for module-local diagnostics; push guards deeper only +on profile evidence (nvtx-on-clocks, `ae67665d3`, gives named `nsys` ranges for free). + +--- + +## 5. SYMPTOM → FIX debugging index + +Consolidated from doc 13 (§1/§1a) plus the mapping/multi-GPU failure modes of docs 02/03/11. +`SILENT` = wrong answer, no crash (the dangerous class). Check row 0 first, always. + +| # | Symptom signature | Root cause | Fix | Anchor | +|---|---|---|---|---| +| 0 | Wrong GPU answers right after a refactor, "looks like a compiler bug" | Your own diff: missing map/copy, misplaced accumulation, unbalanced enter/exit | Audit the diff before blaming nvfortran | `6f3a42d53`, `b404caae2`, `799836a54` | +| 1 | Runtime error / illegal address when an EOS/type-bound method runs in a device loop | Polymorphic `this` v-table dispatch on device | Call a free `_loc` kernel (no `this`); thin host wrapper keeps the API | doc 06; `52a1b3954`, `7c7af5572` | +| 2 | `SILENT` divergence; a `class(...)` actual still appears in a device loop (dead or live) | Residual implicit copy of `this` | Port the remaining call to `_loc`; else accept the copy (open compiler issue) | `MOM_EOS_Wright.F90:1008,1048,1114,1147`; `Roquet_rho:817` | +| 3 | Build error / wrong reduction when target is an array element | nvfortran can't reduce array elements | Stage a named scalar, assign after the loop | `MOM_tracer_hor_diff.F90:962` | +| 4 | Unsupported intrinsic / wrong `modulo()` on device | `modulo()` not implemented on all targets | `sign()` + truncating division (`e - 3*(e/3)` style) | `MOM_intrinsic_functions.F90:232` | +| 5 | `SILENT` last-bit CPU↔GPU diff in `x**(1./n)` | `exp((1/n)*log x)` lowering differs host libm vs libdevice | `cuberoot`/`nth_root` (fixed-iteration Newton, `declare target`) | `MOM_intrinsic_functions.F90:120-132` | +| 6 | `SILENT` wrong result from a specific `do concurrent` (NVHPC 25.9, meridional tracer-flux loop) | Version-specific DC miscompile | Keep it `!$omp target teams loop collapse(2) private(...)` | `MOM_tracer_hor_diff.F90:1464` | +| 7 | `SILENT` wrong answers from early `exit` in a loop nested under DC (NVHPC 25.11) | DC lowering mis-compiles `exit` | Negate condition into an `if`-guarded body; never `exit`/`return`/`cycle` in device loops | `e23d6a7b1` | +| 8 | Correct but slow; kernel launches far fewer teams than expected (17 vs ~238) | Runtime team-count heuristic under-launches | Manual `num_teams(ceiling(iters/128.))` | `5b5f6b2b1`; `MOM_continuity_PPM.F90:707` | +| 9 | `SILENT` wrong numbers when a tiny helper is called from a `target teams`/`loop` region | Un-inlined cross-procedure device call miscompiled | `!DIR$ ATTRIBUTES FORCEINLINE :: name` (or `declare target`) | `3cb184edd`, `93dbbd36e`; `:1086,1149` | +| 10 | Segfault in an `!$omp target`+`parallel loop` region wrapping a k-recurrence | Compiler/runtime bug | Rewrite as `do concurrent` (or teams-loop with serial k) | `5274c3a8e`, `0f05b360f` | +| 11 | Crash on A100 + NVHPC 25.5 at a `map(to: x) if (present(x))` | Conditional map of optional dummy | Delete it; map at the caller | `2108e0eba` | +| 12 | Answers differ **by GPU count** (or run-to-run) — control flow | `map(alloc:)` on a struct whose host-set scalars/`associated()` are read on device (garbage device memory) | `map(to:)` the struct and its member array | `a774eb331`; doc 02 §4b | +| 13 | Answers differ by GPU count — dropped flag/accumulation | Shared-scalar write in DC without `reduce` | `DO_LOCALITY(reduce(: scalar_tmp))`, assign after | `e182de310`; doc 11 §7.1 | +| 14 | Device "addressing error" after init; `associated()` misbehaves in kernels | Parent struct re-`enter data`'d after members were attached | Map parent once; refresh with `update to(parent)`; re-order per doc 02 §5.2 | `c82e1254a` | +| 15 | Checksum "mismatch" that isn't reproducible arithmetic | Stale host copy read by host-only checksum/diag | `update from()` immediately before the call | `b29b27150`; doc 07 §6.4 | +| 16 | GPU time dominated by attach/detach; ~2x slowdown mapping struct arrays | Array-of-structs each holding an array | Flatten to one array with the loop index as a dimension | `1865612de` | +| 17 | Per-call implicit micro-transfers around a derived-type member in an offload pass | Deep-copy trap (`CS%tv%T` through the container) | Explicit `map(to: CS%tv, CS%tv%T, ...)` bracket or bare-pointer alias | `ff86497d5` | +| 18 | Build error on `do concurrent(...) local(...)` on another compiler | Compiler lacks F2023 locality | Wrap every locality clause in `DO_LOCALITY(...)` (`HAVE_FC_DO_CONCURRENT_LOCAL`) | `do_concurrent_compat.h`; `ac/m4/mom6_fc_do_concurrent_local.m4` | +| 19 | Reproducing sum returns NaN on large domains | Size-dependent branch fell back to an unported host routine reading device-only data | Single carry-safe blocked code path for all sizes | `0ac71d482`; doc 07 §3.2 | +| 20 | Bit-repro regression after splitting a producer loop from its accumulation | Compiler re-associates the split reduction (even ifort, even -O0) | Re-fuse producer and consumer loops | `5f413739b`; doc 09 §3 | +| 21 | Device compile failure on automatic arrays sized from non-dummy expressions in `pure`/DC procedures | nvfortran limitation | Fixed-size replacement (`nk=75`, `NK_GPU_MAX=500`) — parameterize before merging | `05c74b56b`; `jorge/diagnostics_port` | +| 22 | Suspected races in a teams-loop kernel; nondeterministic wrong answers | Scalars written in the body missing from `private()`/`local()` | Privatize **every** body-written scalar | `origin/merge-omp-debug` (`7a51e5fb3`) | + +⚠ Row 7's bug shape is only *partially* fixed at HEAD: six sibling early-`exit`-under-DC sites +remain (`MOM_tracer_hor_diff.F90:911,913`; `MOM_tracer_advect.F90:287,292`; +`MOM_vert_friction.F90:700,929`) — treat as latent until checksum-validated under ≥25.11 +(§8, "Six early-`exit`-under-`do concurrent` sites remain at HEAD"). + +NVHPC versions on record: **25.5** (A100 conditional-map crash), **25.9** (DC wrong result), +**25.11** (early-exit miscompile). `__NVCOMPILER_OPENMP_GPU` is the compile-time GPU-build switch +(block-size defaults; disables CPU-only early-exit convergence tests) — flagged in `93dbbd36e` as +"to be replaced at a later time". + +--- + +## 6. PRIORITIZED WORK QUEUE (doc 14 §8, updated with cross-doc context) + +Dependency ordering for Tier 1: **EOS `_loc` coverage → N²/density inputs (`find_N2`, +`isopycnal_slopes`) → `set_diffusivity` → KPP/EPBL → `kappa_shear`**; tridiagonal solves are +independent warm-ups. + +### Tier 1 — unconditional per-step critical path + +1. **`MOM_diabatic_aux.F90` tridiagonals** (`tracer_vertdiff`/`triDiagTS`[`_Eulerian`]). + *Approach:* copy the in-file `find_uv_at_h` template verbatim (teams-loop over `j`, + `do concurrent(i)`, serial `do k`, `map(alloc/release)` scratch). *Hazards:* `tv%T/S` are + pointer members — map the targets; diag posts behind `id>0` updates. *Lowest effort, do first.* +2. **`MOM_set_diffusivity.F90`.** *Precedent:* `edoyango/port-set_diffusivity` has j-blocking + + `(i,j,k)` reorder groundwork (zero device directives yet). *Dependencies:* EOS `_loc` chain and + `MOM_isopycnal_slopes.F90` device coverage first — prefer merging the density-integral work from + `port/thickness_diffuse`. *Hazards:* writes restart-target pointer fields `visc%Kd_*`. +3. **`MOM_energetic_PBL.F90`.** *Precedent:* `origin/epbl-3d` proves whole-column `pure` + + `do concurrent(j,i)` works ("100x… ~2ms/step") but is GPU-only (no CPU-preserving story) and + needed two workarounds to reuse deliberately: manual inlining of + `get_Langmuir_Number`/`find_mstar`, and fixed-size column arrays (parameterize the hardcoded + `nk=75` before merging). Decide naive-vs-blessed explicitly against principle 2. +4. **`MOM_CVMix_KPP.F90`.** Zero in-flight work; same column pattern; *hazard:* calls into external + `pkg/CVMix-src` — the largest `declare target`/inlining surface of Tier 1. +5. **`MOM_kappa_shear.F90`.** Feeds `Kd_shear/Kv_shear`; invoked from the dynamics side + (`set_viscous_ML`), so device data must be live across the dycore→diabatic boundary. Port after + set_diffusivity. + +### Tier 2 — critical in ALE configs / config-conditional + +6. **ALE remap (`MOM_remapping.F90`, `Recon1d_*`)** — the hardest problem (ragged per-column + sizing, `class(Recon1d)` dispatch, deep call chains). *Not a blank slate:* + `origin/jorge/diagnostics_port` tags the whole non-polymorphic OM4 chain `!$omp declare target` + and adds `NK_GPU_MAX=500` fixed sizing (`MOM_remapping.F90:47,1275-1277`) — routines are + device-*callable*, not yet device-*driven*. *Next:* resolve the `NK_GPU_MAX` sizing question (§9), then add the driving + `do concurrent(j,i)` at `MOM_ALE.F90:745`, then bitwise-validate. Treat as research-grade. +7. **`MOM_regridding.F90`** — integer select-case dispatch, no polymorphism; standard template + should apply; zero in-flight work. +8. **`MOM_thickness_diffuse.F90`** — furthest along (`edoyango/port/thickness_diffuse`: real + `do concurrent` + `DO_LOCALITY`, submodule split), but drags in the density-integral/EOS + subsystem (17 files). Merge its EOS/density work early to unblock item 2. +9. **`MOM_mixed_layer_restrat.F90` (Bodner MLE)** — `bodner-naive-port` is the naive contrast case + (doc 10): no block params, many small data regions, three commits to reach `do concurrent`. + Redo blessed-style using doc 10's checklist; reuse its 3-D `calculate_density` call + (`2271af66e`), which is bitwise-safe on the no-`rho_ref` path. +10. **EOS remaining forms** — 7 of 9 unported, **including the default `WRIGHT_FULL`** + (`EOS_DEFAULT`, `MOM_EOS.F90:192`). Mechanical per-form recipe: doc 06 §6.3 (6 steps, one file + per form; do not skip `density_anomaly`). Also `int_spec_vol_dp_wright` (non-Boussinesq path) + is unported. + +### Tier 3 — supporting/infra + +11. **Diagnostics mediator** — continue `diag_map_mediator_port` (transfer elimination for + mask/downsample/conversion paths; the FMS write path stays host). 12. **k-block completion** — + land `kblock-hor-visc` (watch the declaration-order CPU-perf lesson) and the remaining + CorAdCalc TODO bodies (OBC/WENO). 13. **btstep tuning** — evaluate the `acc-btstep` async + hypothesis (§9, "single-stream serialization") before investing. 14. **Restart path** — keep host-only (`noport` + class); enforce the dominating-transfer rule instead. 15. **Port-coverage tooling** — adopt + `edoyango/gpu-port-tracking` (`.testing/tools/track_gpu_port.py`, `!@start noport/toport` + sentinels) as the objective progress metric. + +--- + +## 7. PROVEN-WORKS inventory and NEVER-DO list + +### 7.1 Proven to work (each with its merged evidence) + +1. Mapping allocatable arrays inside derived types with co-located + `enter/exit data` next to `ALLOC_`/`DEALLOC_` (`MOM_dynamics_split_RK2.F90:1350-1368`). +2. Whole-CS shells mapped `alloc` before child `_init`; nested sub-type shell-then-members + (`MOM.F90:3225-3231`); whole-struct `update to(CS)` for batch scalar refresh + (`MOM_barotropic.F90:6576`). +3. k-blocking with `#ifdef __NVCOMPILER_OPENMP_GPU` 0-vs-tile defaults, bitwise-verified — + continuity (`93dbbd36e`), CorAdCalc (`b8c471cfa`); the hybrid tiled + `target teams num_teams` + `loop collapse(2)` kernel (`MOM_continuity_PPM.F90:696-738`). +4. `do concurrent` + `DO_LOCALITY(local/local_init/reduce)` as the compute idiom, with the + configure-time compatibility macro (698 uses; doc 04). +5. Teams-loop `collapse(2)` + serial-k tridiagonal columns with `declare target` column kernels + (`MOM_vert_friction.F90:737`, `find_uv_at_h`). +6. Block-based EFP reproducing sums on GPU — exact-integer `reduce(+:block_sum)` partitioned into + carry-safe blocks, bit-identical under any scheduling (`8593a732a`; `MOM_coms.F90:618-772`). +7. GPU-aware grouped halo exchange: `do_group_pass(..., omp_offload=.true.)` at 26 sites, + replacing manual staging (`656e09013`); wide-halo BT_Domain march-inward amortization. +8. EOS `_loc` free-function + explicit `do concurrent` 2D/3D overrides (Wright, Roquet_rho; + `7c7af5572`); host-resolved v-table, device `_loc` execution (`2271af66e` reuse in MLE). +9. `pure` + `!$omp declare target` helpers returning error flags via arguments (`efp_decompose`); + `cuberoot`/`nth_root` bit-stable intrinsic replacements. +10. Guarded transfers (`update from(...) if (id>0 .or. debug)`, `if (allocated/associated)` maps); + blanket state transfer at sync points (`MOM.F90:1091`). +11. Struct-of-arrays flattening for attach-cost (halved GPU time, `1865612de`). +12. nvtx-on-clocks profiling wrapper — every existing `cpu_clock` becomes a named `nsys` range with + zero call-site changes (`ae67665d3`, branch). +13. Multi-GPU-correct tracer advection (scalar-temp reductions + `map(to:)` registry; + `e182de310`/`a774eb331`). + +### 7.2 NEVER-DO list + +1. **Never reorder floating-point arithmetic** — no re-association, no split producer/reduction + loops, no distributing parentheses (`5f413739b`; Fortran parens pin evaluation order). +2. **Never pass polymorphic `this`/`class(*)` into a device region**, or dispatch through a + v-table on device (`MOM_EOS_Wright.F90:1008` comments; doc 06 §1.4). +3. **Never `map(alloc:)` a struct whose host-set contents are read on device** (`a774eb331`). +4. **Never re-`enter data` a parent struct after its members are attached** (`c82e1254a`). +5. **Never write a shared scalar/array element from a DC body without `reduce`** — and the reduce + target must be a scalar (`e182de310`; `MOM_tracer_hor_diff.F90:962`). +6. **Never use early `exit`/`return`/`cycle` inside device-offloaded loop bodies** (`e23d6a7b1`). +7. **Never `map(...) if (present(optional_arg))` inside a callee** (`2108e0eba`). +8. **Never call a helper from a `target teams`/`loop` region without guaranteed inline or + `declare target`** — silent wrong numbers (`3cb184edd`). +9. **Never use `modulo()` or `x**(1./n)` in device code where bits matter** (rows 4/5 of §5). +10. **Never allocate, do I/O, or call `post_data` inside a device loop.** +11. **Never mix guard intrinsics** (`associated` on an allocatable or vice versa) (doc 02 §3). +12. **Never map arrays-of-structs element-by-element in a loop** on a hot path (`1865612de`). +13. **Never leave an enter-data without its mirrored exit-data** (`15ca2a25f`), never expect + `delete`/`release` to copy back, and **never `map(delete:)` an object your scope does not + own** — it zeroes the refcount and destroys outer mappings (§8, "The `ADp` mapping lifecycle"). +14. **Never ship a port without the bitwise checksum gate and a ≥2-GPU run** (§3 Step 9). +15. **Never hand-roll a reproducible float sum** — use `reproducing_sum`/EFP (doc 07 §6.1). + +--- + +## 8. Verified findings + +Settled by a source-and-git review pass (2026-07-14; no builds or runs). State these as fact; do not +re-derive them. Anything still genuinely unknown lives in §9. + +**The `ADp` mapping lifecycle is internally inconsistent.** At HEAD, `initialize_MOM` (`MOM.F90`) +maps `CS%ADp` persistently with `enter data map(alloc:)` — refcount 1, device copy a garbage shell. +The first `vertvisc` (`MOM_vert_friction.F90`) call does `enter data map(to: ADp)`; it is already +present, so the refcount goes to 2 and **the `to` copy is skipped**, leaving the shell garbage. Only +the explicitly attach-mapped `du_dt_str`/`dv_dt_str` get valid device descriptors — the sole reason +the device-side `associated(ADp%…)` reads in `vertvisc` are safe. The matching +`exit data map(delete: ADp)` then **forces the refcount to 0**, destroying `initialize_MOM`'s +mapping; every later `vertvisc` call re-creates the shell fresh, now with a real `to` copy. Net: the +init-time map is dead weight that suppresses the first call's shell refresh and is then silently +destroyed. Fix (maintainer's choice): either drop the init-time map and let `vertvisc` own the +per-call lifecycle with `release`, or make the init-time map authoritative (`map(to:)` + per-call +`update to(ADp)`, no per-call delete). Do **not** `map(to:)` the shell in `initialize_MOM` — that +was proposed and is wrong. + +**A `map(to:)` on an already-present object does not refresh device contents.** If a struct's host +scalars or descriptors changed since its first map, the only refresh is `target update to(...)`. +Several existing patterns rely on this implicitly; new code must never "re-map to refresh". + +**`delete` vs `release` is load-bearing, in the dangerous direction.** `exit data map(delete:)` +forces the refcount to zero, so a per-call `delete` inside a callee destroys any outer, persistent +mapping of the same object — as `vertvisc`'s delete kills `initialize_MOM`'s `ADp` map on the first +call. Rule: `release` for scoped/per-call teardown; `delete` only in the owning `*_end` routine that +mirrors the owning `enter data`. Never `map(delete:)` an object your scope does not own. + +**"Partial presence" is the literal NVIDIA runtime diagnostic.** The NVHPC OpenMP/OpenACC runtime +raises a FATAL "partially present" error when a mapping's address range partially overlaps an +existing present-table entry — exactly the whole-struct-over-attached-member overlap the docs +inferred. Rely on it. + +**`c82e1254a`'s root cause was re-allocated storage, not a refcount subtlety.** The host +re-allocated `CS%visc`, so the second `map(to:)` targeted *different* storage than the first, +orphaning member attachments. The "map the parent exactly once" rule guards against this regardless +of which reading of the OpenMP spec you take. + +**The `GV` device map is load-bearing, not vestigial.** `GV%Rlay` is read inside a device +`do concurrent` — the `Rml_max`-vs-`GV%Rlay` binary density search in `tracer_epipycnal_ML_diff` +(`MOM_tracer_hor_diff.F90`) — so `initialize_MOM`'s `map(to: GV, GV%Rlay, GV%g_prime)` is consumed. + +**Six early-`exit`-under-`do concurrent` sites remain at HEAD; `e23d6a7b1` fixed only one.** NVHPC +25.11 produced wrong answers from an early `exit` in a loop nested inside a DC (never-do #6), yet the +identical shape survives in `tracer_epipycnal_ML_diff` (`MOM_tracer_hor_diff.F90`, the binary-search +`exit`s — the *same subroutine* as the fixed insert-sort), `advect_tracer` (`MOM_tracer_advect.F90`, +the `domore` search loops), and `vertvisc` (`MOM_vert_friction.F90`, the `direct_stress` column +loops, which also contain the device-side `associated(ADp%…)` reads). The ban is **empirical per +NVHPC version**, not structural — doc 04 §5.5 branch 5 cites an acceptable serial-k early-exit inside +a DC, which is why the knowledge base previously contradicted itself here. Until each site is +checksum-validated under ≥25.11, treat all six as latent wrong-answer bugs and apply the `e23d6a7b1` +if-guard rewrite opportunistically. (`direct_stress` and `tracer_epipycnal_ML_diff` are non-default +code paths, which is likely why nothing has tripped.) + +**Rejecting an array-element `reduce` is conforming F2023, not an nvfortran quirk.** A +locality-spec/reduce list takes *variable names*; `max_srt(j)` is an array element, not a variable. +Whole-array `reduce(+: block_sum)` is conforming and positively supported. The staged-scalar +workaround stays correct. + +**"Inline or wrong answers" has a primary source: `3cb184edd`'s commit body** — "for OpenMP, +inlining of ratio_max and flux_elem is MANDATORY … Otherwise results are incorrect." This is +era-specific evidence (the OpenACC→OpenMP translation, pre-`num_teams`-fix kernel), not a timeless +law — but treat it as binding for new code. + +**`ratio_max`'s missing directive is a deliberate removal resting on implicit device codegen.** +`93dbbd36e` removed `!NVF$ INLINE` from `ratio_max` without replacement (while giving +`flux_elem`/`flux_elem_OBC` `FORCEINLINE`), and no `-Minline` exists in any in-repo or mkmf-template +build config. `ratio_max` is still called from `!$omp target`/`loop` regions in +`MOM_continuity_PPM.F90`. Correctness at HEAD therefore rests on nvfortran implicitly +compiling/inlining a small same-file `pure` function for the device — empirically fine on the tested +toolchain (the commit is merged and checksum-gated), but fragile. **Recommendation:** add +`!DIR$ ATTRIBUTES FORCEINLINE :: ratio_max` for parity; never imitate the gap in new code. + +**The Wright anomaly `this` branch is mainline-safe but a live hazard on the pf branch.** On +`dev/gpu`, the generic 2D/3D-plus-`rho_ref` dispatch is reached only from host paths. On +`port/pressureforce-benchmark_ALE`, the k-blocked `int_density_dz_generic_plm` +(`MOM_density_integrals.F90`) calls 3-D `calculate_density(..., rho_ref=rho_ref)` with +`use_rho_ref = .true.` **by default**, dispatching into the `present(rho_ref)` branch that passes +polymorphic `this` inside a `do concurrent` (`calculate_density_array_2d_buggy_Wright` and its 3-D +sibling, `MOM_EOS_Wright.F90`). **Merge gate for that branch:** add +`density_anomaly_elem_buggy_Wright_loc` first — trivial, and Roquet proves the pattern. + +**FMA contraction is pinned in the canonical NVHPC toolchain.** `mkmf/templates/ncrc5-nvhpc.mk` (and +`ncrc-nvhpc.mk`) put `-Mnofma` (plus `-Mdaz`) in the **base** `FFLAGS`, for all build modes. Action: +ensure the site GPU build harness (external to this repo) inherits `-Mnofma`. If it does, CPU↔GPU +bit-identity does not depend on the two toolchains happening to contract identically. + +**FMS `omp_offload` is a genuine device path with NO fallback.** In the sibling FMS checkout, +`mpp_group_update.fh` device-packs halos (`target teams distribute … if(use_device_ptr)` into a +device buffer) and `mpp_transmit_mpi.fh` posts `MPI_ISEND`/`IRECV` inside +`!$omp target data use_device_ptr(...)` — real CUDA-aware MPI on device pointers. There is **no** +capability check: a non-GPUDirect MPI stack means crash or corruption, not graceful host staging. The +nonblocking variants hardcode `use_device_ptr = .false. ! placeholder`, confirming doc 11's +gated/unconditional analysis from the FMS side. + +**Restart staleness is latent, not live.** `save_MOM_restart` (`MOM.F90`) does no transfer of its +own, but `step_MOM`'s sync-point blanket `update from(u, v, h, CS%uhtr, CS%vhtr)` runs whenever +`MOM_state_is_synchronized(CS)` — the same condition under which the driver writes restarts — and +thermo/mixing fields are host-authoritative (diabatic is host-only). Standing rule: any *newly +device-resident* restart-registered field must be added to a dominating `update from` before +`save_restart`. + +**"We lose present()" (`f74525ae8`) means the Fortran intrinsic, not the OpenACC data clause.** The +diff contains no OpenACC `present()` data clauses anywhere, but it does contain commented-out OpenMP +maps of the form `!!!$omp target enter data if(present(pbce)) map(to: pbce)` — the author tried +intrinsic-`present()` conditional maps and disabled them. The foreshadowing link to the `2108e0eba` +A100 crash (the same construct) stands. + +--- + +## 9. Open questions + +None of these are answerable from source or git. Each needs a run, a profile, or a maintainer's +decision — go straight to the stated experiment rather than re-deriving from source. + +**Needs a run or a profile:** + +- **`do concurrent` unspecified-locality semantics.** F2018 leaves locality *unspecified* by default; + nvfortran documents privatizing scalars whose first access in the construct is a write. The docs' + caution stands; the decisive check is `-Minfo=accel` output on one kernel, not more source reading. +- **`do concurrent` single-stream serialization.** NVHPC's documented model launches DC kernels on + the default CUDA stream per host thread, so serialization of independent kernels is *expected* — + which is exactly what `acc-btstep`'s `async(1..3)` queues attack. Confidence is high (documented + behaviour), but quantify with one `nsys` timeline of btstep before investing (Tier-3 item 13). +- **`NK_GPU_MAX=500` sizing.** At `GV%ke≈75`, 500-deep per-thread private column arrays over-allocate + device local memory ~6.7×; several such arrays per thread will spill and crush occupancy. Prefer + sizing from the dummy argument (`size(h,3)`) or a blocked redesign. The underlying constraint + (nvfortran rejecting non-dummy-sized automatics in device `pure` procedures, `05c74b56b`) is real, + so a `parameter` sized to a realistic maximum (e.g. 128) plus an init-time `FATAL` guard is the + pragmatic middle. Needs an occupancy measurement to settle. (Gates Tier-2 item 6.) + +**Needs a maintainer decision:** + +- **PLM density-integral team launch.** Does the PLM hot path need continuity's manual + `num_teams(ceiling(...))` workaround, or does the tile geometry here (a `5*TILE_SIZE_X` inner + dimension) keep nvfortran's default team launch adequate? Compare any surviving `target teams loop` + in `int_density_dz_generic_plm`/`PressureForce_FV_Bouss` on `port/pressureforce-benchmark_ALE` + against the under-launch symptom that motivated `5b5f6b2b1`. +- **Which PLM branch is merge-ready.** Is `port/pressureforce-benchmark_ALE` genuinely more + merge-ready than the naive port, or merely *different*? Its only edges are the `0x1` CPU default + and the `desubmodule`. Whether `0x1` beats `32x4` on CPU, and whether desubmoduling is the intended + end-state, needs a benchmark and a maintainer call. +- **`diag_map_mediator_port` caller-residency audit** — the gate before merging that branch. The + rewritten `diag_remap_calc_hmask`/`downsample_*` routines assume their array arguments are already + device-resident and do no transfer themselves. Confirm every caller establishes that residency: a + caller handing in a host-only array would read uninitialized device memory silently. Check that the + `h` argument threaded into `diag_remap_calc_hmask` is mapped at every call site, not just the mask. +- **`NONBLOCKING_UPDATES` policy for GPU production runs.** 14 of 26 GPU-aware halo sites silently + revert to host-staged communication when it is enabled (doc 11 §4). If GPU configs are expected to + run with it off, document that (and consider asserting at init when `__NVCOMPILER_OPENMP_GPU` + builds detect it on); if on, the 14 gated sites are a standing performance trap. Needs a param-doc + note. +- **The EOS endgame.** Continue the per-form `_loc` boilerplate for the remaining 7 forms (including + the default `Wright_full`), or adopt the polymorphism-free `select case (form_of_EOS)` dispatch + sketched — as a proposal only — in doc 06 §6.4? The residual `this`-descriptor copy is structural + under the current design; the `select case` route removes it once for all forms. Note that the + Wright-anomaly finding above makes the `_loc` route costlier than doc 06 estimated, since the + anomaly kernels must be duplicated too. Upstream PR #156 / the `eos-3d` branches may already answer + this — check before investing in 7 more `_loc` conversions. (Shapes Tier-2 item 10.) + +--- + +## 10. Index of `docs/gpu-knowledge/` + +| Doc | One line | +|---|---| +| `00-architecture.md` | Anchor: layout, CS pattern, step_MOM/RK2 call tree, memory model, merged-work inventory (patched 2026-07-14 to drop its stale `!NVF$ INLINE` story and counts) | +| `01-memory-control-structures.md` | Memory macros, symmetric-memory mechanics, full CS type/allocation graph, the 3 shaping commits (`81680c15d`/`c82e1254a`/`1865612de`) | +| `02-pointer-usage.md` | Pointer taxonomy, `associated()` map guards, the `Reg%Tr(:)` and `visc` mapping bugs, pointer-hazard table | +| `03-openmp-mapping.md` | Proven mapping patterns: lifecycle, shells, conditional maps, map-kind table, declare-target catalogue, update-from taxonomy | +| `04-do-concurrent-patterns.md` | DC census (698), `DO_LOCALITY` machinery, locality-specifier catalogue, DC-vs-teams evidence, the loop-form decision tree | +| `05-kblocking-tiling.md` | The blessed transform: before/after diffs (continuity/CorAdCalc/hor_visc), block-size machinery, bitwise argument, 11-step recipe | +| `06-eos-layer.md` | Old polymorphic dispatch, `_loc` rewrite, per-form port-status table, 6-step port-a-form recipe, endgame proposal | +| `07-reproducibility.md` | EFP algorithm + GPU blocking, purity rules, checksum tooling, order-of-operations rulebook | +| `08-cross-module-inlining.md` | Device-callable helper catalogue, inline-directive history (3 forms), refused constructs, device-call checklist | +| `09-barotropic-solver.md` | btstep structure, wide-halo march-inward, repro/crash fixes, acc-async experiment, transferable lessons | +| `10-inflight-ports.md` | Naive (bodner) vs blessed (pf-ALE) contrast, branch forensics, 8-gate merge review rubric | +| `11-halos-domains-multigpu.md` | Group-pass machinery, `omp_offload` end-to-end, 26-site table, gated/unconditional split, multi-GPU bug anatomy, halo porting rules | +| `12-diagnostics-io.md` | Host-only mediator/restart, transfer audit (guarded vs blanket), diag_map_mediator_port scope, nvtx profiling, restart rules | +| `13-compiler-workarounds.md` | The 24-row bug/workaround catalogue, symptom-signature index, NVHPC version table, directive/flag reference | +| `14-vertical-physics-ale-status.md` | Diabatic/ALE status tables, column-kernel anatomy, in-flight branch survey, tiered remaining-work plan | diff --git a/knowledge/README.md b/knowledge/README.md new file mode 100644 index 0000000..4266d34 --- /dev/null +++ b/knowledge/README.md @@ -0,0 +1,40 @@ +# MOM6 GPU knowledge base + +The knowledge the `skills/` in this repo read from. `KNOWLEDGE.md` is the operational playbook +(porting procedure, decision rules, symptom→fix index, prioritized work queue, proven-works and +never-do lists); `gpu-knowledge/00-14` are the deep-dive docs behind it, plus standalone bug notes. + +Read `KNOWLEDGE.md` first — it is self-contained, and every load-bearing rule is inlined with its +citation so you can act without opening the deep docs. + +## Provenance — read this before trusting a line number + +The knowledge base cites MOM6 source by `file:line`, plus commit hashes and branch names. Those are +only meaningful against a specific tree: + +| | | +|---|---| +| `file:line` refs valid against | `dev/gpu` @ **`c82e1254a`** | +| upstream baseline | `dev-gfdl` @ `c3237e27f` | +| built | 2026-07-14, from source + git only — no builds or runs | + +**Line numbers rot**, and they rot invisibly here: there is no MOM6 source in this repo to +contradict a stale reference. Verify any anchor before acting on it. Commit hashes and branch names +do not rot — prefer them where both are available. + +## The confidence markers are load-bearing + +`KNOWLEDGE.md` §8 and §9 record, per claim, which were **verified from source**, which are +**advanced but need a run or a profile to close**, and which are **open maintainer decisions**. The +same distinction appears in the bug notes. That grading is the most valuable thing in here — please +preserve it when editing rather than flattening everything to assertion. + +Several findings are explicitly **unconfirmed** and say so. They are recorded because they are +worth checking, not because they are established. + +## Status + +Current best practice, not a finished spec. The port is a work in progress and only the code paths +exercised by the `benchmark` / `benchmark_ALE` configurations have been validated. When you hit a +pattern these docs do not cover, make a decision consistent with the philosophy, flag it, and add it +here once it is confirmed. diff --git a/knowledge/gpu-knowledge/00-architecture.md b/knowledge/gpu-knowledge/00-architecture.md new file mode 100644 index 0000000..85f71f7 --- /dev/null +++ b/knowledge/gpu-knowledge/00-architecture.md @@ -0,0 +1,416 @@ +# MOM6 Architecture for GPU Porting (dev/gpu) + +> **Note (2026-07-14):** this anchor document predates the verification pass over docs 01–14. +> Known-stale spots have been corrected inline; where this file and a numbered doc disagree, +> **the numbered doc and `KNOWLEDGE.md` win.** + +> **Purpose.** This is the anchor document for the MOM6 GPU-porting knowledge base. It describes the +> code architecture a porting agent must understand *before* touching anything: directory layout, +> the control-structure (CS) pattern, the time-stepping call tree, memory conventions, and a +> quantified inventory of GPU work already merged on `dev/gpu`. Every other document in +> `docs/gpu-knowledge/` drills into one subsystem; read this one first. +> +> **Repo state at time of writing.** Branch `dev/gpu`, upstream baseline `dev-gfdl`. Diffing +> `dev-gfdl...dev/gpu` = 320 commits, 52 files, +8310/−4615 lines = the totality of merged GPU work. +> The port targets NVIDIA GPUs with **NVHPC / nvfortran** using **OpenMP target offload** + +> **Fortran `do concurrent`**. Do not build or run the code; study source + git only. + +--- + +## 0. Guiding principles of the port (context that shapes every decision) + +1. **Preserve CPU performance.** The blessed strategy is **k-blocking / tiling**: loops are + restructured into blocks (`niblock`/`njblock`/`nkblock`) so one source form runs well on both + CPU (cache blocking, block sizes `32/4/1`) and GPU (whole-array, block sizes `0/0/0`). See + merged commits `b8c471cfa` (Kblock coradcalc), `93dbbd36e` (k-block continuity), and local branch + `kblock-hor-visc`. +2. **Bitwise reproducibility is mandatory.** Refactors must not reorder floating-point operations. + Extracting code into `pure`/`elemental` subroutines is the preferred restructuring tool. + Reproducing sums use exact fixed-point integer arithmetic (`MOM_coms`) so results are independent + of thread/PE order. +3. **`do concurrent` is the default parallel idiom.** OpenMP `target teams` directives are used only + for reductions or where `do concurrent` misbehaves/underperforms (documented per case). +4. **Cross-module calls inside device loops are painful.** They must be force-inlined or duplicated + as `!$omp declare target` helpers. (CORRECTION: at HEAD the live directive is + `!DIR$ ATTRIBUTES FORCEINLINE`; `!NVF$ INLINE` and `-Minline=name:` are historical — + `93dbbd36e`; see doc 08.) **`class(*)`/runtime polymorphism is a disaster on device** — the EOS + layer is being rewritten to avoid it. +5. **Document nvfortran bugs.** This is frontier work; genuine compiler bugs are hit and worked + around with comments and directives that must be catalogued. + +--- + +## 1. Directory and module layout + +Source lives under `src/` (~244 F90 files) with the FMS coupling/infra under `config_src/`. + +| Directory | Role | Key modules | +|---|---|---| +| `src/core/` | Dynamical core (momentum + continuity), prognostic state, grids | `MOM.F90`, `MOM_dynamics_split_RK2.F90`, `MOM_continuity_PPM.F90`, `MOM_CoriolisAdv.F90`, `MOM_PressureForce_FV.F90`, `MOM_barotropic.F90`, `MOM_variables.F90`, `MOM_grid.F90`, `MOM_verticalGrid.F90`, `MOM_open_boundary.F90` | +| `src/parameterizations/lateral/` | Lateral (horizontal) subgrid physics | `MOM_hor_visc.F90`, `MOM_thickness_diffuse.F90`, `MOM_mixed_layer_restrat.F90`, `MOM_MEKE.F90`, `MOM_lateral_mixing_coeffs.F90` | +| `src/parameterizations/vertical/` | Vertical physics / mixing | `MOM_vert_friction.F90`, `MOM_set_viscosity.F90`, `MOM_diabatic_driver.F90`, `MOM_set_diffusivity.F90`, `MOM_CVMix_KPP.F90`, `MOM_energetic_PBL.F90`, `MOM_kappa_shear.F90` | +| `src/ALE/` | Vertical Lagrangian remap (ALE), regridding, 1-D reconstructions | `MOM_ALE.F90`, `MOM_regridding.F90`, `MOM_remapping.F90`, `PPM/PLM/PQM_functions.F90`, `Recon1d_*.F90`, `coord_*.F90` | +| `src/equation_of_state/` | Equation of state (density) | `MOM_EOS.F90`, `MOM_EOS_base_type.F90`, `MOM_EOS_Wright.F90`, `MOM_EOS_Roquet_rho.F90`, others | +| `src/tracer/` | Tracer registry, advection, diffusion | `MOM_tracer_registry.F90`, `MOM_tracer_types.F90`, `MOM_tracer_advect.F90`, `MOM_tracer_hor_diff.F90` | +| `src/diagnostics/` | Runtime diagnostics, energy/mass integrals | `MOM_diagnostics.F90`, `MOM_sum_output.F90` | +| `src/framework/` | Infrastructure: domains, comms, IO, params, diag mediator, checksums | `MOM_domains.F90`, `MOM_coms.F90`, `MOM_diag_mediator.F90`, `MOM_restart.F90`, `MOM_file_parser.F90`, `MOM_hor_index.F90`, `MOM_checksums.F90`, `MOM_intrinsic_functions.F90`, `do_concurrent_compat.h`, `MOM_memory_macros.h` | +| `src/initialization/` | State/grid initialization | `MOM_state_initialization.F90` | +| `config_src/infra/{FMS1,FMS2}/` | Thin wrappers over GFDL FMS (mpp domains, IO, diag manager) | `MOM_domain_infra.F90`, `MOM_diag_manager_infra.F90`, `MOM_coms_infra.F90` | +| `config_src/memory/` | Compile-time memory model selection | `dynamic_symmetric/MOM_memory.h`, `dynamic_nonsymmetric/MOM_memory.h` | +| `config_src/drivers/` | Top-level drivers (solo, coupled) | `solo_driver/MOM_driver.F90` | + +**Physics/infra split to keep in mind:** the *dynamical core* (`src/core`) is where the bulk of the +GPU port has happened; the *vertical physics* (`diabatic`, `set_diffusivity`, KPP, EPBL) and *ALE +remapping/regridding* are largely **untouched on `dev/gpu`** and in-flight on side branches (see §6). + +--- + +## 2. The control-structure (CS) pattern + +Every module owns exactly one derived type `_CS` (declared `type, public :: X_CS ; private` — +public name, private members) that holds **all persistent per-module state**: runtime parameters, +diagnostic IDs (init to −1), work arrays, halo group-pass handles, and pointers to child CSs. It is +populated by a `_init` routine and torn down by `_end`. + +### 2.1 Member styles + +Members come in three flavours; which one is used has direct GPU-mapping consequences: + +- **Macro-allocatable arrays** via `MOM_memory_macros.h`, e.g. in `MOM_dynamics_split_RK2.F90:90`: + `real ALLOCABLE_, dimension(NIMEMB_PTR_,NJMEM_,NKMEM_) :: CAu, PFu, diffu`. `ALLOCABLE_` expands to + `,allocatable`; `ALLOC_(x)` to `allocate(x)`. This is the dominant, GPU-friendly form. +- **Bare `pointer` arrays**, e.g. `real, pointer, dimension(:,:) :: taux_bot => NULL()` + (`MOM_dynamics_split_RK2.F90:151`). Pointers are used mainly so an array can be a **target in the + restart registry** (`MOM_variables.F90:294` comment) or aliased across modules. Pointers complicate + device mapping and aliasing analysis. +- **Nested child CSs**, in two idioms: *by value/embedded* (`type(hor_visc_CS) :: hor_visc`, + `type(continuity_CS) :: continuity_CSp`, `MOM_dynamics_split_RK2.F90:244,246`) or *by pointer* + (`type(vertvisc_CS), pointer :: vertvisc_CSp => NULL()`, line 252). + +### 2.2 The nesting tree + +`MOM_control_struct` (`MOM.F90:204`) is the root, allocated once. It holds the prognostic state as +macro-allocatables (`h, T, S` at `MOM.F90:205`; `u, uh, uhtr`; `v, vh, vhtr`), the grid/vertical-grid +pointers (`G`, `GV`, `US`), shared containers (`tv`, `visc`, `ADp`, `CDp`), and the child module CSs. +The dynamics hub is `dyn_split_RK2_CSp` (pointer, `MOM.F90:407`), which in turn embeds +`hor_visc`, `continuity_CSp`, `CoriolisAdv`, `barotropic_CSp` (by value) and points to +`vertvisc_CSp`, `set_visc_CSp`, `ALE_CSp`. + +``` +MOM_control_struct (MOM.F90:204) ← allocated once, mapped alloc at MOM.F90:3xxx +├── prognostic state: h,T,S,u,v,uh,vh,uhtr,vhtr (macro-allocatable, mapped `to`) +├── G / GV / US (grid; G%* metrics uploaded in bulk) +├── tv (thermo_var_ptrs, allocatable) ← T,S pointers + eqn_of_state +├── visc (vertvisc_type, allocatable) +├── ADp / CDp (accel_/cont_diag_ptrs, pointer aliases) +└── dyn_split_RK2_CSp (pointer, MOM.F90:407) ← the dycore hub CS + ├── CAu,PFu,diffu,eta,u_av,... (macro-allocatable 3D) ← enter/exit data by member + ├── hor_visc (continuity_CS, by value) + ├── continuity_CSp (continuity_CS, by value; params only, block sizes) + ├── CoriolisAdv (CoriolisAdv_CS, by value) + ├── barotropic_CSp (barotropic_CS, by value; large frhatu/frhatv/... arrays) + ├── vertvisc_CSp (vertvisc_CS, pointer) + └── set_visc_CSp (set_visc_CS, pointer) +``` + +### 2.3 Shared "bag of state" containers (`MOM_variables.F90`) + +These are threaded through many CSs so several modules alias the same fields: + +- `thermo_var_ptrs` (`:79`) — pointer `T`, `S`, `p_surf`, `frazil`, allocatable `SpV_avg`, and + `type(EOS_type), pointer :: eqn_of_state`. +- `ocean_internal_state` (`:138`) — all-pointer aliases to `T,S,u,v,h,uh,vh` and accelerations. +- `accel_diag_ptrs` (`:167`) / `cont_diag_ptrs` (`:241`) — pointer diagnostic aliases (`CS%ADp`,`CS%CDp`). +- `vertvisc_type` (`:258`) — **hybrid**: allocatable drag fields (`bbl_thick_u`, `kv_bbl_u`, `Ray_u`) + plus pointer fields (`MLD`, `Kd_shear`, `Kv_shear`) that must be restart-registry targets. +- `BT_cont_type` (`:317`) — all-allocatable barotropic face-area coupling arrays. + +**GPU implication:** the offload unit is the *whole CS object with its allocatable members*. Deep-copy +("attach/detach") of derived-type member arrays on device is expensive; commit `1865612de` flattened +arrays-of-structs to flat arrays because time was wasted attaching member arrays per struct +(halved GPU time in `MOM_tracer_hor_diff`). + +--- + +## 3. Memory conventions + +### 3.1 Compile-time memory model — `config_src/memory/` + +Two dynamic configs on the include path differ by exactly one `#define`: +`dynamic_symmetric/MOM_memory.h` defines `SYMMETRIC_MEMORY_`, `dynamic_nonsymmetric` undefs it. Both +undef `STATIC_MEMORY_` (so `dev/gpu` uses dynamic allocation). Halo width `NIHALO_ = NJHALO_ = 2`. +Static memory would substitute real `NIGLOBAL_`/`NK_` at build time. + +### 3.2 Macros — `src/framework/MOM_memory_macros.h` + +- Attribute macros: `ALLOCABLE_`→`,allocatable`, `PTR_`→`,pointer`, `ALLOC_(x)`→`allocate(x)`, + `DEALLOC_(x)`, `TO_NULL_`→`=>NULL()` (all no-ops in static mode). +- Heap-shape macros (dynamic): `NIMEM_`,`NJMEM_`→`:`; the **B (velocity/corner) forms depend on + symmetric memory** — `NIMEMB_`/`NJMEMB_`→`0:` if symmetric else `:`; `NIMEMB_SYM_`→`0:` always. +- Dummy-argument shape macros (the `SZ*` family, appear in nearly every subroutine signature): + `SZI_(G)`→`G%isd:G%ied`, `SZJ_(G)`→`G%jsd:G%jed`, `SZK_(G)`→`G%ke`, `SZK0_(G)`→`0:G%ke`, + `SZIB_(G)`→`G%IsdB:G%IedB`, `SZJB_(G)`→`G%JsdB:G%JedB`. + +### 3.3 Index conventions — `src/framework/MOM_hor_index.F90` / `MOM_grid.F90` + +`hor_index_type` (`MOM_hor_index.F90:18`) scalar integers, replicated into `ocean_grid_type` +(`MOM_grid.F90:28`): + +- Cell-center (h/tracer point): `isc/iec`, `jsc/jec` (computational); `isd/ied`, `jsd/jed` (data = + computational + halos); `isg/ieg` (global). +- Cell-vertex / velocity B-grid (capital-I/J "B"): `IscB/IecB`, `JscB/JecB`; `IsdB/IedB`, `JsdB/JedB`. +- In **symmetric** mode the B indices start one lower: `IsdB = isd-1` (`MOM_hor_index.F90:92-96`), + matching `NIMEMB_`→`0:`. Upper bounds always equal center upper bounds. + +Canonical array shapes (doc block `MOM_hor_index.F90:178`): `h(isd:ied,jsd:jed)`, +`q(IsdB:IedB,JsdB:JedB)`, `u(IsdB:IedB,jsd:jed)`, `v(isd:ied,JsdB:JedB)`. Grid metrics are named by +stagger: `T`=tracer/h, `Cu`=C-grid u, `Cv`=C-grid v, `Bu`=B-grid corner; reciprocals prefixed `I` +(`IareaT`). **Code loop ranges should always be written for symmetric memory** — non-symmetric then +also works (with a less efficient halo pattern). + +### 3.4 Where the state lives + +Prognostic arrays `h,T,S` (h/tracer points), `u,uh,uhtr` (u points), `v,vh,vhtr` (v points) are +macro-allocatables inside `MOM_control_struct` (`MOM.F90:205-216`). They are passed by argument down +the call tree (as `u,v,h`), aliased into `tv%T`, `tv%S`, and into the `ocean_internal_state` pointer +container for diagnostics. The vertical grid (`GV%ke` layers, `GV%sInterface`, unit factors) is a +`pointer` carried everywhere. + +--- + +## 4. Time-stepping call tree (the GPU-critical path) + +### 4.1 Top level — `MOM.F90` `step_MOM` + +`step_MOM` (`MOM.F90:522`) drives a coupling timestep. Within it (order depends on ALE/thermo +splitting flags): + +- `step_MOM_thermo` (`MOM.F90:911, 1032`; def `:1731`) → `diabatic(...)` (`:1828`) → ALE remap. +- `step_MOM_dynamics` (`MOM.F90:985`; def `:1226`) → `step_MOM_dyn_split_RK2(...)` (`:1388`) [or the + `RK2b`/unsplit variants]. +- `step_MOM_tracer_dyn` (`MOM.F90:999`; def `:1598`) → `advect_tracer(...)` (`:1650`) → + `tracer_hordiff(...)` (`:1653`). +- Diagnostics posted throughout via `post_data` (`MOM_diag_mediator.F90`). + +Group halo update of the coupled state uses the **GPU-aware** path: +`call do_group_pass(pass_uv_T_S_h, G%Domain, ..., omp_offload=.true.)` (`MOM.F90:2112`). + +### 4.2 The split RK2 dynamical core — `MOM_dynamics_split_RK2.F90` + +`step_MOM_dyn_split_RK2` (def `:302`) is a **predictor–corrector** scheme separating fast barotropic +and slow baroclinic modes. Actual internal call sequence (absolute line numbers in +`MOM_dynamics_split_RK2.F90`; group passes created at `:506-517`): + +**Predictor stage** +1. `PressureForce(h,tv,...)` → `CS%PFu,CS%PFv,CS%pbce,CS%eta_PF` (`:527`) +2. `CorAdCalc(u_av,v_av,h_av,...)` → `CS%CAu_pred,CS%CAv_pred` (`:589`) — Coriolis + momentum advection +3. `set_viscous_ML(...)` (`:640`), `vertvisc_coef(up,vp,...)` (`:650`), `vertvisc_remnant(...)` (`:651`) +4. `btcalc(h)` (`:671`), `bt_mass_source` (`:673`) +5. `continuity(u_inst,v_inst,h,hp,...,BT_cont)` → provisional `hp` (`:695`); `set_dtbt` (`:715/:719`) +6. **`btstep(...)`** — barotropic sub-cycling (`:726`), returns `u_accel_bt,v_accel_bt,eta_pred` +7. `vertvisc_coef` (`:801`) / `vertvisc(up,vp,...,AD_pred)` (`:817`) / `vertvisc_remnant` (`:834`) +8. `continuity(up,vp,h,hp,...,u_cor=u_av,v_cor=v_av)` (`:853`); `radiation_open_bdry_conds` (`:868`) + +**Corrector stage** +9. `bt_mass_source(hp,...)` (`:894`), `PressureForce(hp,...)` (`:909`, if `begw/=0`) +10. `horizontal_viscosity(u_av,v_av,h_av,...)` → `CS%diffu,CS%diffv` (`:962`) +11. `CorAdCalc(...)` → `CS%CAu,CS%CAv` (`:972`) +12. `btstep(...)` again (`:1023`) +13. `vertvisc_coef(u_inst,v_inst)` (`:1099`) / `vertvisc(...,ADp)` (`:1108`) / `vertvisc_remnant` (`:1121`) +14. `continuity(u_inst,v_inst,h_tmp,h,...)` → final `h` (`:1148`); `radiation_open_bdry_conds` (`:1172`) +15. Optional stored `CorAdCalc(...,CAu_pred,CAv_pred)` for next predictor (`:1206`) + +Group-pass execution points: `pass_eta` (`:580/:658`), `pass_visc_rem` (`:661-685`), `pass_uvp` +(`:829-847`), `pass_hp_uv` (`:860`), `pass_vector(u_av,v_av)` OBC (`:877`), `pass_uv` (`:1118-1137`), +`pass_h` (`:1153`), `pass_av_uvh` (`:1163-1181`) — all `omp_offload=.true.`. **The pure compute +kernels (continuity_PPM, CorAdCalc, PressureForce_FV, hor_visc) contain no halo updates and no +reproducing sums by design** — communication is hoisted into this driver and `btstep`. + +So the per-timestep dycore hot loop is: **PressureForce → CorAdCalc → vert_visc(coef/remnant) → +btcalc → continuity → btstep (inner barotropic loop) → hor_visc → continuity**, with ~10 group halo +updates, all on the `omp_offload=.true.` path. + +### 4.3 Key sub-solvers + +- **`continuity` / `MOM_continuity_PPM.F90`** — PPM finite-volume mass transport. Public `continuity` + → `zonal_mass_flux` / `meridional_mass_flux` → PPM reconstruction. **Fully k-blocked** with + `niblock/njblock/nkblock` (CS members `:76-78`), hybrid `do concurrent` + `!$omp target teams` (see §5). +- **`CorAdCalc` / `MOM_CoriolisAdv.F90`** — Coriolis + advection of momentum (KE gradient, vorticity + flux). Heavily restructured (56 `do concurrent`, 61 `omp target`); k-blocking merged (`b8c471cfa`). +- **`PressureForce` / `MOM_PressureForce_FV.F90`** — finite-volume pressure gradient. Two entry points + `PressureForce_FV_nonBouss` (`:122`) and `PressureForce_FV_Bouss` (`:947`); calls into + `MOM_density_integrals.F90` (`int_density_dz_*`) which call EOS. 30 `do concurrent`, 34 `omp target`. +- **`btstep` / `MOM_barotropic.F90`** — barotropic solver (`:480`, 6868-line module). Sub-cycles many + small barotropic timesteps in `btstep_timeloop` (`:2376`); `set_dtbt` (`:3797`) sets the sub-step. + Most heavily ported module by raw directive count (242 `do concurrent`, 106 `omp target`). +- **`horizontal_viscosity` / `MOM_hor_visc.F90`** — lateral friction (Laplacian + biharmonic, + Smagorinsky/Leith). 62 `do concurrent`, 144 `omp target`. k-blocking in-flight on `kblock-hor-visc`. +- **`vertvisc` / `MOM_vert_friction.F90`** — implicit vertical friction (tridiagonal solve per column). + `vertvisc_coef`/`vertvisc`/`vertvisc_remnant`. Uses `!$omp target teams loop collapse(2)` with a + serial inner tridiagonal k-loop (see §5). 3 `!$omp declare target` column kernels. +- **`diabatic` / `MOM_diabatic_driver.F90`** — vertical mixing dispatcher (`diabatic` `:279` → + `diabatic_ALE`/`layered_diabatic`). **Essentially unported on `dev/gpu`** (3-line change). + +--- + +## 5. The k-blocking / tiling transformation (the blessed pattern) + +The core CPU-and-GPU-preserving refactor. Loops are rewritten so a horizontal/vertical **block** is +processed at a time. Block sizes are CS parameters resolved at init: on NVHPC GPU builds they default +to `0` (meaning "whole domain / no cache blocking"), on CPU to e.g. `32/4/1` +(`MOM_continuity_PPM.F90:3120-3129`). When `0`, the block is set to the full loop extent +(`if (niblock == 0) niblock = ...`, `MOM_continuity_PPM.F90:188`). + +Representative hybrid kernel from continuity (`MOM_continuity_PPM.F90:696-736`): an outer host loop +strides over blocks, block-local indices `ii = i-i_start+1`, `jj = j-j_start+1` reuse small work +arrays, and the compute region is an explicit `!$omp target teams num_teams(nteams)` (team count +computed by hand — `nteams = ceiling(real((j_end-j_start+1)*(i_end-i_start+1))/128.)`) wrapping +`!$omp loop collapse(2)` over the tile, calling `!$omp declare target` helpers (`flux_elem`, +`ratio_max`). The manual team count exists because nvfortran's OpenMP runtime under-launched teams +(commit `5b5f6b2b1`). Elsewhere the same file uses plain `do concurrent (k=1:nz, j=..., i=...)` +(`:430`) where the compiler schedules acceptably. + +Bitwise preservation: k-blocking changes only *loop structure*, never the arithmetic order within a +column reconstruction, so results stay bit-identical — verified with `MOM_checksums`. Merged examples: +`93dbbd36e` (continuity reconstruction), `b8c471cfa` (CoriolisAdv). In-flight: `kblock-hor-visc` +(single commit rewrites `MOM_hor_visc.F90`, +1365/−1113), `kblock-coradcalc`. + +--- + +## 6. Quantified inventory of GPU work + +### 6.1 Merged on `dev/gpu` (`git diff --stat dev-gfdl...dev/gpu`, 320 commits, 52 files) + +**Ported (heavy churn, GPU-resident):** + +| Subsystem | File | +/− | Status | +|---|---|---|---| +| Continuity (PPM) | `MOM_continuity_PPM.F90` | +1575/−1171 | k-blocked, hybrid dc+omp | +| Barotropic solver | `MOM_barotropic.F90` | +1003/−804 | ported; `omp_offload` halos | +| Vertical friction | `MOM_vert_friction.F90` | +775/−184 | teams-loop tridiagonal | +| Coriolis/advection | `MOM_CoriolisAdv.F90` | +764/−565 | k-blocked (`b8c471cfa`) | +| Set viscosity (BBL/ML) | `MOM_set_viscosity.F90` | +501/−384 | ported, declare-target kernels | +| Horizontal viscosity | `MOM_hor_visc.F90` | +399/−195 | ported (full k-block on branch) | +| EOS dispatch | `MOM_EOS.F90` + base/Wright/Roquet | +361/+154/+290/+291 | Wright+Roquet 2D/3D direct | +| Tracer hor. diff | `MOM_tracer_hor_diff.F90` | +319/−268 | ported; flat-array refactor | +| Tracer advection | `MOM_tracer_advect.F90` | +319/−227 | ported (`#54`); multi-GPU fixes | +| Reproducing sums | `MOM_coms.F90` | +317/−177 | block-based EFP (`8593a732a`) | +| Pressure force FV | `MOM_PressureForce_FV.F90` | +215/−136 | ported (Wright integrals) | +| Dycore driver | `MOM_dynamics_split_RK2.F90` | +244/−105 | maps CS members, halo offload | +| Top driver | `MOM.F90` | +260/−40 | top-level CS/grid mapping | +| Intrinsics | `MOM_intrinsic_functions.F90` | +104/−23 | `cuberoot`/`nth_root` declare-target | + +Infra added: `do_concurrent_compat.h` (`DO_LOCALITY` macro), `ac/m4/mom6_fc_do_concurrent_local.m4` ++ `ac/configure.ac:172` (feature detection → `HAVE_FC_DO_CONCURRENT_LOCAL`), `omp_offload` optional +arg on `do_group_pass` (`config_src/infra/FMS2/MOM_domain_infra.F90:1143`). + +**Directive totals across `src/`:** 698 `do concurrent`, 829 `omp target`, 213 `enter data`, +168 `exit data`, 21 `declare target`. (Counts re-verified by doc 03's verification pass.) + +### 6.2 In-flight on branches + +| Branch | Scope | Pattern | +|---|---|---| +| `kblock-hor-visc` | `MOM_hor_visc.F90` k-blocking | blessed k-block | +| `kblock-coradcalc` | `MOM_CoriolisAdv.F90` k-block (subset merged) | blessed k-block | +| `bodner-naive-port` | `MOM_mixed_layer_restrat.F90` (Bodner MLE), density integrals | **naive** port (contrast case) | +| `port/pressureforce-benchmark_ALE` | `MOM_density_integrals.F90` int_density_dz PLM | ALE pressure integrals | +| `diag_map_mediator_port` | `MOM_diag_mediator.F90` (+365) | diagnostics offload | +| `fix/nan_repro_sum` | `MOM_coms.F90` NaN in large-domain repro sum | reproducibility bugfix | +| `remotes/edoyango/acc-btstep` | `MOM_barotropic.F90` OpenACC kernels + async | alternative btstep offload | +| `remotes/edoyango/bugfix-traceradvection-multigpu` | `MOM_tracer_advect.F90` `Reg%Tr(:)` mapping | multi-GPU correctness | +| `remotes/edoyango/port-set_diffusivity` | `MOM_set_diffusivity.F90` (+1277) | vertical-mixing port | +| `remotes/edoyango/port/thickness_diffuse` | `MOM_thickness_diffuse.F90` (+685) | lateral-mixing port | +| `remotes/edoyango/gpu-port-tracking` | `.testing/tools/track_gpu_port.py` + CI | **port coverage tooling** (noport/toport sentinels) | +| `remotes/edoyango/benchmark_ALE_nvtx_clocks` | nvtx markers on clocks | profiling practice | + +### 6.3 Untouched on `dev/gpu` (host-only; major porting surface remaining) + +`MOM_diabatic_driver.F90` (3 lines), `MOM_set_diffusivity.F90` (0), `MOM_CVMix_KPP.F90` (0), +`MOM_energetic_PBL.F90` (0), `MOM_mixed_layer_restrat.F90` (0), `MOM_ALE.F90` (3), +`MOM_regridding.F90` (0), `MOM_remapping.F90` (0), `MOM_diag_mediator.F90` (0, host-only IO path). +The **entire vertical-mixing (diabatic) stack and ALE remap/regrid remain to be ported.** + +--- + +## 7. Cross-cutting subsystems a porting agent must know + +### 7.1 EOS — runtime polymorphism is the enemy + +`EOS_type` (`MOM_EOS.F90:117`) wraps a **polymorphic allocatable component** +`class(EOS_base), allocatable :: type` (`:162`). `EOS_base` (`MOM_EOS_base_type.F90:13`) is an abstract +type with **deferred elemental type-bound procedures** (`density_elem`, etc.). The concrete class is +chosen with `allocate( :: EOS%type)` in a `select case`/`select type` (`MOM_EOS.F90:2122`), +and calls dispatch through `EOS%type%calculate_density_...`. **nvfortran cannot resolve this v-table +dispatch on device, and passing polymorphic `this` into a `do concurrent`/target region forces a +mishandled implicit copy.** The fix (merged, `7c7af5572`, `52a1b3954`): each elemental kernel is +duplicated as a free `_loc` function with no `this`, and new `calculate_density_array_3d` / +`_derivs_3d` / `_second_derivs_2d` implementations wrap `do concurrent (k,j,i)` calling the `_loc` +kernel. **Only buggy-Wright and Roquet_rho are ported;** linear, UNESCO, Jackett06, TEOS10, +Wright_full/red, Roquet_SpV still fall back to polymorphic elemental dispatch. See `06-eos.md`. + +### 7.2 Bitwise reproducibility — `MOM_coms` EFP sums + `MOM_checksums` + +Global integrals (`write_energy`, `MOM_sum_output.F90`) use **Extended Fixed Point (EFP)** reproducing +sums: each real is decomposed into a 6-word base-2⁴⁶ signed integer (`EFP_type`, `MOM_coms.F90:103`), +integers are summed exactly (associative regardless of order), and reconstructed. The GPU port +(`8593a732a`) partitions the domain into **blocks small enough that no block sum can overflow the 17 +carry bits**, and each block is a `do concurrent` with `DO_LOCALITY(reduce(+: block_sum))` + +`reduce(max:...)` over exact integers (`increment_block_ints`, `MOM_coms.F90:618-772`), with +`efp_decompose` a `pure`/`!$omp declare target` helper (`:778`). Because the reduction is over exact +integers, thread/PE scheduling cannot change the bits. `MOM_checksums.F90` verifies ports via a +`popcnt`-based bitcount checksum (`:2680`) mod 10⁹ — two runs agree only if fields are bit-identical. +See `07-reproducibility.md`. + +### 7.3 Halos, domains, and the `omp_offload` path + +`MOM_domains.F90` re-exports `create_group_pass`/`do_group_pass`/`pass_var`/`pass_vector` over FMS mpp +(`config_src/infra/FMS2/MOM_domain_infra.F90`). The GPU port added an optional `omp_offload` argument +to `do_group_pass` (`:1143`) forwarded to `mpp_do_group_update`, so halo exchanges operate on +device-resident buffers (GPU-aware). It is passed `.true.` at 26 call sites (14 gated behind +`if (G%nonblocking_updates)`, 12 unconditional — doc 11 §4) across dynamics, +barotropic, and tracer modules. Halo width is 2 (`NIHALO_`). See `11-halos-domains.md`. + +### 7.4 Diagnostics / IO — still host-only on `dev/gpu` + +`MOM_diag_mediator.F90` (`post_data` generic, `:73`) is **unchanged on `dev/gpu`** — posting a +diagnostic implies a device→host transfer of the field. Directives guard transfers behind diag-ID +checks (`!$omp target update from(...) if (CS%id_... > 0)`, e.g. `MOM_tracer_hor_diff.F90:722`). The +offload of the mediator itself is on branch `diag_map_mediator_port`. See `12-diagnostics-io.md`. + +### 7.5 Compiler workarounds + +Catalogue-worthy so far: mandatory inlining of `ratio_max`/`flux_elem` (`3cb184edd`: "Otherwise +results are incorrect"; historically via `-Minline=name:`/`!NVF$ INLINE`, at HEAD via +`!DIR$ ATTRIBUTES FORCEINLINE` on flux_elem/flux_elem_OBC only — `93dbbd36e`; `ratio_max` currently +carries no directive, see KNOWLEDGE.md §8, "`ratio_max`'s missing directive"), `!$omp declare target` on all point/column +kernels, manual `num_teams` (`5b5f6b2b1`), +`omp target teams loop -> do concurrent` reversions (`e8b0ecfbf`), `modulo()` avoided in `cuberoot` +(not implemented on all targets), EOS `_loc` free functions to avoid `this` copies, "implicit copy of +`this` which cannot yet be prevented" (unresolved), and the `A100 nvfortran 25.5` crash avoided by +removing an `eta_bt` transfer (`2108e0eba`). See `13-compiler-workarounds.md`. + +--- + +## 8. Port-coverage tooling (branch `edoyango/gpu-port-tracking`) + +`.testing/tools/track_gpu_port.py` cross-references gcov execution coverage against auto-detected +ported regions (`do concurrent`, `!$omp target[ teams][ loop]` blocks). Manual overrides use in-source +sentinels: `!@start noport ... !@end noport` (never port — serial bookkeeping) and +`!@start toport ... !@end toport` (needs porting but not a structural loop). Every marker requires an +explicit matching `!@end`. Executed-but-unported lines are split into "portable" vs "not portable" +(allocate/IO/call/control-flow). This is the objective progress metric for the whole effort. + +--- + +## 9. Quick reference — where to look first + +- **Add a device kernel:** copy the `do concurrent (k,j,i) DO_LOCALITY(local(...))` pattern; for + reductions use `DO_LOCALITY(reduce(+:...))`; for tridiagonal columns use + `!$omp target teams loop collapse(2)` with explicit `private`. +- **Map a new CS array:** `ALLOC_(CS%x(...)); CS%x=0.0; !$omp target enter data map(to: CS%x)` and the + mirrored `map(delete:)` next to `DEALLOC_` in `*_end`. +- **Call a helper from device:** add `!$omp declare target` and ensure it inlines. +- **Verify a port:** compare `MOM_checksums` hchksum/uchksum and reproducing-sum energy output CPU vs GPU. +- **Never** pass `class(*)`/polymorphic `this`, allocate inside a device loop, or reorder a + floating-point reduction. diff --git a/knowledge/gpu-knowledge/01-memory-control-structures.md b/knowledge/gpu-knowledge/01-memory-control-structures.md new file mode 100644 index 0000000..afb1e93 --- /dev/null +++ b/knowledge/gpu-knowledge/01-memory-control-structures.md @@ -0,0 +1,836 @@ +# Memory Handling and Control-Structure Allocation (dev/gpu) + +> Companion to `00-architecture.md` §2 (CS pattern) and §3 (memory conventions). Where +> `00-architecture.md` sketches the CS pattern and macro table, this document goes to the bottom of +> it: exact macro expansions, the concrete symmetric/non-symmetric offset mechanics, full type +> definitions for the five+ study-target CS types with allocation-site line numbers, and a +> line-by-line reconstruction of how the CS object graph is built and mapped onto the GPU at runtime. +> See `02-pointer-usage.md` for the deep dive on *why* individual fields are `pointer` (restart +> registry, cross-module aliasing) — this document only touches that where it bears on allocation. + +--- + +## 1. The compile-time memory model + +### 1.1 `config_src/memory/` — the one-`#define` difference + +Two files select the memory layout at build time; both are otherwise byte-identical: + +- `config_src/memory/dynamic_symmetric/MOM_memory.h:37`: `#define SYMMETRIC_MEMORY_` +- `config_src/memory/dynamic_nonsymmetric/MOM_memory.h:37`: `#undef SYMMETRIC_MEMORY_` + +Both files (`:41`) `#undef STATIC_MEMORY_` — `dev/gpu` always builds dynamic (heap-allocated, +runtime-shaped) arrays, never static (compile-time-shaped) arrays. Both set `NIHALO_ = NJHALO_ = 2` +(`:30,33`) and `NIGLOBAL_`/`NJGLOBAL_`/`NK_`/`NIPROC_`/`NJPROC_` to `NONSENSE_*` placeholders +(`:11-21`) that are never actually used in dynamic mode (they only matter for `STATIC_MEMORY_` +builds, where they're substituted by the build system with real numbers). Both `#include +` (`:43`) which does the real work, branching on whether `STATIC_MEMORY_` is +defined. + +### 1.2 `src/framework/MOM_memory_macros.h` — full macro catalogue (dynamic-mode expansions) + +Attribute/action macros (`:101-110`, identical role in both modes, only dynamic shown): +| Macro | Expands to (dynamic) | Static-mode value | +|---|---|---| +| `ALLOCABLE_` | `,allocatable` | *(nothing)* | +| `PTR_` | `,pointer` | *(nothing)* | +| `ALLOC_(x)` | `allocate(x)` | *(nothing)* | +| `DEALLOC_(x)` | `deallocate(x)` | *(nothing)* | +| `TO_NULL_` | `=>NULL()` | *(nothing)* | + +Heap-shape macros for declaring `ALLOCABLE_`/`PTR_` members inside a type (dynamic mode, `:114-161`): +| Macro | Dynamic expansion | Notes | +|---|---|---| +| `NIMEM_` / `NJMEM_` | `:` | h/tracer-point extent | +| `NIMEMB_PTR_` / `NJMEMB_PTR_` | `:` (**always**, regardless of symmetric) | see note below | +| `NIMEMB_` / `NJMEMB_` | `0:` if `SYMMETRIC_MEMORY_` else `:` (`:126-140`) | velocity/corner (B) extent | +| `NIMEMB_SYM_` / `NJMEMB_SYM_` | `0:` unconditionally | *always*-symmetric B arrays | +| `NKMEM_` | `:` | layer extent | +| `NKMEM0_` | `0:` | interface extent | +| `NK_INTERFACE_` | `:` | interface extent (heap-shape macro, `NK_+1` in static mode) | + +Dummy-argument / stack-shape macros, the `SZ*` family used in nearly every subroutine signature +(`:167-182`): +``` +SZI_(G) -> G%isd:G%ied SZJ_(G) -> G%jsd:G%jed +SZK_(G) -> G%ke SZK0_(G) -> 0:G%ke +SZIB_(G) -> G%IsdB:G%IedB SZJB_(G) -> G%JsdB:G%JedB +SZIBS_(G) -> G%isd-1:G%ied SZJBS_(G)-> G%jsd-1:G%jed ! "always symmetric" dummy shape +``` +Plus decomposition-invariant `SZDI_/SZDIB_/SZDJ_/SZDJB_` (`:188-195`) that are the same in both memory +models — used where a routine must always see the symmetric-shaped index range regardless of the +build's `SYMMETRIC_MEMORY_` setting. + +**The `NIMEMB_PTR_` subtlety.** In *dynamic* mode `NIMEMB_PTR_`/`NJMEMB_PTR_` expand to plain `:` +(`MOM_memory_macros.h:122,125`) — **not** `0:` — even when `SYMMETRIC_MEMORY_` is set. This is +because in dynamic mode the array is declared with assumed/deferred shape (`:`); the actual lower +bound (`0` vs `1`) is fixed later at the `ALLOC_(...)` call site using explicit bounds computed from +`hor_index_type`/`ocean_grid_type` (`IsdB`, which is itself `isd-1` when symmetric — see §2). In +*static* mode, by contrast, `NIMEMB_PTR_` is literally `NIMEMB_` (`MOM_memory_macros.h:53,56`) +because the shape has to be baked into the declaration at compile time — there's no later `ALLOC_` +call to fix it up. So: **dynamic-mode symmetric offsets live in the runtime `ALLOC_` bounds, not in +the type declaration**; static-mode symmetric offsets live in the macro expansion itself. + +### 1.3 Concrete example — `ocean_grid_type` in `MOM_grid.F90` + +```fortran +! MOM_grid.F90:78-92 (h-point vs u-point metric declarations) +real ALLOCABLE_, dimension(NIMEM_,NJMEM_) :: & + mask2dT, geoLatT, geoLonT, dxT, IdxT, dyT, IdyT, areaT, IareaT, sin_rot, cos_rot +real ALLOCABLE_, dimension(NIMEMB_PTR_,NJMEM_) :: & + mask2dCu, OBCmaskCu, geoLatCu, geoLonCu, dxCu, IdxCu, IdxCu_OBCmask, dyCu, IdyCu, dy_Cu, IareaCu, areaCu +``` +In dynamic mode both declarations reduce to `dimension(:,:)` — the difference between an h-point +array (`NIMEM_`) and a u-point array (`NIMEMB_PTR_`) disappears at declaration time and is entirely +determined by the bounds passed to `ALLOC_` in `allocate_metrics` (`MOM_grid.F90:536-617`): +```fortran +! MOM_grid.F90:547-548 +ALLOC_(G%dxT(isd:ied,jsd:jed)) ; G%dxT(:,:) = 0.0 +ALLOC_(G%dxCu(IsdB:IedB,jsd:jed)) ; G%dxCu(:,:) = 0.0 +``` +`IsdB` here is `G%IsdB`, computed in `MOM_grid_init` (`:303-313`, mirroring `hor_index_init`): +```fortran +! MOM_grid.F90:306-310 +if (G%symmetric) then + G%IscB = G%isc-1 ; G%JscB = G%jsc-1 + G%IsdB = G%isd-1 ; G%JsdB = G%jsd-1 + G%IsgB = G%isg-1 ; G%JsgB = G%jsg-1 +endif +``` +So **symmetric memory gives every B-staggered (u/v/corner) array one extra row/column on the west/ +south side** (`IsdB = isd-1` instead of `isd`), so that `u(IsdB:IedB, jsd:jed)` includes the west +face of the westernmost h-cell — every h-cell's both faces are present. Non-symmetric memory keeps +`IsdB = isd` (velocity data domain same size as tracer data domain), which is smaller and requires a +different (more argument-passing-heavy, "get the extra column from a neighbor") halo-update pattern +that is legal but less efficient — hence the doc-comment "code should always be written for +symmetric memory" (`MOM_hor_index.F90:175-176`). + +### 1.4 `src/framework/MOM_hor_index.F90` — where the offset is actually computed + +`hor_index_type` (`:18-57`) is the single source of truth for these bounds; `hor_index_init` +(`:65-102`) sets the B (capital-letter) bounds from the h/tracer bounds: +```fortran +! MOM_hor_index.F90:89-99 +HI%IscB = HI%isc ; HI%JscB = HI%jsc +HI%IsdB = HI%isd ; HI%JsdB = HI%jsd +HI%IsgB = HI%isg ; HI%JsgB = HI%jsg +if (HI%symmetric) then + HI%IscB = HI%isc-1 ; HI%JscB = HI%jsc-1 + HI%IsdB = HI%isd-1 ; HI%JsdB = HI%jsd-1 + HI%IsgB = HI%isg-1 ; HI%JsgB = HI%jsg-1 +endif +HI%IecB = HI%iec ; HI%JecB = HI%jec +HI%IedB = HI%ied ; HI%JedB = HI%jed +HI%IegB = HI%ieg ; HI%JegB = HI%jeg +``` +Upper bounds (`IecB`/`IedB`/`IegB`) are **always** equal to the center upper bounds — only the lower +bound moves. `ocean_grid_type` (`MOM_grid.F90:28-216`) duplicates every one of these fields as its own +scalars (`G%isc`, `G%IsdB`, ...) rather than referencing `G%HI` directly in hot code, because `G%HI` +(a `hor_index_type` value, `MOM_grid.F90:31`) is copied wholesale in `MOM_grid_init` (`G%HI = HI`, +`:271`) and the flat scalars are what every `SZI_`/`SZIB_` macro and every loop bound actually reads. + +Canonical shapes (doc block, `MOM_hor_index.F90:178-182`): +``` +h(isd:ied, jsd:jed) q(IsdB:IedB, JsdB:JedB) +u(IsdB:IedB, jsd:jed) v(isd:ied, JsdB:JedB) +``` + +### 1.5 Static-mode expansions (for contrast; not used on `dev/gpu`) + +If `STATIC_MEMORY_` were defined, `NIMEM_` → `(((NIGLOBAL_-1)/NIPROC_)+1+2*NIHALO_)` +(`MOM_memory_macros.h:31`) — a compile-time arithmetic expression substituting real +`NIGLOBAL_`/`NIPROC_`/`NIHALO_` values baked in by the build system (`NIHALO_=2` always) — and +`NKMEM_` → `NK_` (a literal layer count, `:65`), so every array gets a fixed shape at compile time. +`dev/gpu` never uses this path (`STATIC_MEMORY_` is `#undef`'d in both memory configs), but static +mode is why the `SZ*` macros exist at all: dummy-argument declarations must work identically whether +the actual heap array behind them is shaped by a runtime `G%isd` or a compile-time arithmetic +expression. + +--- + +## 2. `verticalGrid_type` (`MOM_verticalGrid.F90:26-101`) + +Unlike `ocean_grid_type`, the vertical grid has almost no macro-allocatable arrays — it is dominated +by scalars (unit-conversion factors `H_to_m`, `H_to_kg_m2`, `Angstrom_H`, ...) plus four small, +**plain** (non-macro) `allocatable` 1-D arrays sized by `nk` (number of layers), not by horizontal +extent: +```fortran +! MOM_verticalGrid.F90:41-46, 63-65 +real, allocatable, dimension(:) :: sLayer !< layer-center coordinate values +real, allocatable, dimension(:) :: sInterface !< interface coordinate values +real, allocatable, dimension(:) :: g_prime, Rlay +``` +`GV` itself is a `pointer` in every owner (`MOM_control_struct%GV`, `MOM.F90:240-241`), allocated +once by `verticalGridInit`: +```fortran +! MOM_verticalGrid.F90:122-124, 242-245 +if (associated(GV)) call MOM_error(FATAL, 'verticalGridInit: called with an associated GV pointer.') +allocate(GV) +... +allocate( GV%sInterface(nk+1) ) +allocate( GV%sLayer(nk) ) +allocate( GV%g_prime(nk+1), source=0.0 ) +allocate( GV%Rlay(nk), source=0.0 ) +``` +and torn down by `verticalGridEnd` (`:361-368`, `deallocate(GV%g_prime, GV%Rlay)` then +`deallocate(GV%sInterface, GV%sLayer)` then `deallocate(GV)`). Because the vertical grid is tiny +(`O(nk)`, not `O(ni*nj*nk)`), most of its data reaches device kernels by-value — scalar +unit-conversion factors passed through argument lists, or the whole small `GV` derived type passed as +an `intent(in)` dummy that nvfortran can firstprivate-copy. + +**Correction (verified against source).** It is *not* true that `GV` never appears in an `!$omp target +enter data` list. `MOM.F90:3650` contains an **active** directive: +```fortran +! MOM.F90:3650 +!$omp target enter data map(to: GV, GV%Rlay, GV%g_prime) +``` +i.e. the whole `GV` object plus its `Rlay` and `g_prime` member arrays are explicitly mapped `to` the +device. Instructively, an earlier attempt to do the same map immediately after `verticalGridInit` +(`MOM.F90:3053-3054`) is **commented out** with the note explaining why it was moved: +```fortran +! MOM.F90:3056-3057 +! This does not work. GV%RLay changes sometime later. +!!!$omp target enter data map(to: GV, GV%Rlay, GV%g_prime) +``` +So the live map was deliberately relocated to a point (`:3650`, after the vertical grid parameters are +rescaled) where `GV%Rlay`/`GV%g_prime` no longer change on the host. `GV%sLayer`/`GV%sInterface` are +*not* in any device map — only `GV`, `Rlay`, and `g_prime`. + +> **Resolved (2026-07-14):** The `GV` device map is load-bearing, not vestigial. `GV%Rlay` is read +> inside a device `do concurrent` — the `Rml_max`-vs-`GV%Rlay` binary density search in +> `tracer_epipycnal_ML_diff` (`MOM_tracer_hor_diff.F90`) — so `initialize_MOM`'s +> `map(to: GV, GV%Rlay, GV%g_prime)` is genuinely consumed. Treat the relocated map as required, and +> keep the ordering constraint the disabled sibling records: it must stay after the host-side rescale. + +--- + +## 3. The five-plus CS types: definitions and allocation sites + +### 3.1 `MOM_control_struct` (`MOM.F90:204-477`) — the root + +Not a pointer or allocatable itself — it is a **plain value type** owned by the driver +(`config_src/drivers/solo_driver/MOM_driver.F90:174`: `type(MOM_control_struct) :: MOM_CSp`) and +passed down by `intent(inout)`. This matters enormously for GPU mapping — see §5. + +Prognostic state is macro-allocatable, declared directly in the type: +```fortran +! MOM.F90:205-216 +real ALLOCABLE_, dimension(NIMEM_,NJMEM_,NKMEM_) :: h, T, S +real ALLOCABLE_, dimension(NIMEMB_PTR_,NJMEM_,NKMEM_) :: u, uh, uhtr +real ALLOCABLE_, dimension(NIMEM_,NJMEMB_PTR_,NKMEM_) :: v, vh, vhtr +``` +allocated in `initialize_MOM` at `MOM.F90:3110-3115`: +```fortran +ALLOC_(CS%u(IsdB:IedB,jsd:jed,nz)) ; CS%u(:,:,:) = 0.0 +ALLOC_(CS%v(isd:ied,JsdB:JedB,nz)) ; CS%v(:,:,:) = 0.0 +ALLOC_(CS%h(isd:ied,jsd:jed,nz)) ; CS%h(:,:,:) = GV%Angstrom_H +ALLOC_(CS%uh(IsdB:IedB,jsd:jed,nz)) ; CS%uh(:,:,:) = 0.0 +ALLOC_(CS%vh(isd:ied,JsdB:JedB,nz)) ; CS%vh(:,:,:) = 0.0 +!$omp target enter data map(to: CS%u, CS%v, CS%h, CS%uh, CS%vh) +``` +and deallocated in `MOM_end` (`:4778`: `DEALLOC_(CS%u) ; DEALLOC_(CS%v) ; DEALLOC_(CS%h)`; the +`MOM_end` subroutine itself begins at `:4698`, and calls `end_dyn_split_RK2(CS%dyn_split_RK2_CSp)` at +`:4733`). + +Nested/embedded children (excerpt; the type has ~30 child-CS members): +```fortran +! MOM.F90:233-234, 244, 261, 340, 342, 403-460 (representative) +type(ocean_grid_type), allocatable :: G_in !< allocatable (was plain value; see 81680c15d, §6) +type(ocean_grid_type), pointer :: G => NULL() !< pointer alias to the active grid +type(thermo_var_ptrs), allocatable :: tv +type(vertvisc_type), allocatable :: visc !< allocatable (was plain value; see 81680c15d/c82e1254a, §6) +type(accel_diag_ptrs), allocatable :: ADp +type(cont_diag_ptrs) :: CDp !< embedded value (not allocatable/pointer) +type(MOM_dyn_split_RK2_CS), pointer :: dyn_split_RK2_CSp => NULL() +type(set_visc_CS), allocatable :: set_visc_CSp !< allocatable (was plain value; see c82e1254a, §6) +type(thickness_diffuse_CS) :: thickness_diffuse_CSp !< embedded value +type(MEKE_CS) :: MEKE_CSp !< embedded value +``` +Allocation sites for the notable ones: `allocate(CS%tv)` (`:2564`), `allocate(CS%G_in)` (`:2985`, +followed by `!$omp target enter data map(to: CS%G_in)` at `:3034`), `allocate(CS%ADp)` (`:3189`, +`!$omp target enter data map(alloc: CS%ADp)` `:3190`), `allocate(CS%visc)` (`:3277`, `map(alloc: +CS%visc)` `:3278`), `allocate(CS%dyn_split_RK2_CSp)` (`:3258`, `map(alloc: CS%dyn_split_RK2_CSp)` +`:3259`). + +### 3.2 `MOM_dyn_split_RK2_CS` (`MOM_dynamics_split_RK2.F90:89-281`) — the dycore hub + +Pointer member of the parent (`CS%dyn_split_RK2_CSp`, `MOM.F90:407`), allocated at `MOM.F90:3258` (see +above) — **the CS type itself has no `_init`-time `allocate(CS)` inside its own module**; the parent +allocates it because "this module does not have its own control structure, but shares the same +control structure with MOM.F90" (module doc, `MOM_dynamics_split_RK2.F90:2113-2116`). + +Array members are macro-allocatable: +```fortran +! MOM_dynamics_split_RK2.F90:90-95 +real ALLOCABLE_, dimension(NIMEMB_PTR_,NJMEM_,NKMEM_) :: & + CAu, CAu_pred, PFu, PFu_Stokes, diffu +``` +Allocated (with the GPU-map immediately following each pair) in `register_restarts_dyn_split_RK2`, +`:1350-1368`: +```fortran +! MOM_dynamics_split_RK2.F90:1350-1368 +ALLOC_(CS%diffu(IsdB:IedB,jsd:jed,nz)) ; CS%diffu(:,:,:) = 0.0 +ALLOC_(CS%diffv(isd:ied,JsdB:JedB,nz)) ; CS%diffv(:,:,:) = 0.0 +!$omp target enter data map(to: CS%diffu, CS%diffv) +ALLOC_(CS%CAu(IsdB:IedB,jsd:jed,nz)) ; CS%CAu(:,:,:) = 0.0 +ALLOC_(CS%CAv(isd:ied,JsdB:JedB,nz)) ; CS%CAv(:,:,:) = 0.0 +!$omp target enter data map(to: CS%CAu, CS%CAv) +ALLOC_(CS%CAu_pred(IsdB:IedB,jsd:jed,nz)) ; CS%CAu_pred(:,:,:) = 0.0 +ALLOC_(CS%CAv_pred(isd:ied,JsdB:JedB,nz)) ; CS%CAv_pred(:,:,:) = 0.0 +!$omp target enter data map(to: CS%CAu_pred, CS%CAv_pred) +ALLOC_(CS%PFu(IsdB:IedB,jsd:jed,nz)) ; CS%PFu(:,:,:) = 0.0 +ALLOC_(CS%PFv(isd:ied,JsdB:JedB,nz)) ; CS%PFv(:,:,:) = 0.0 +!$omp target enter data map(to: CS%PFu, CS%PFv) +ALLOC_(CS%eta(isd:ied,jsd:jed)) ; CS%eta(:,:) = 0.0 +ALLOC_(CS%u_av(IsdB:IedB,jsd:jed,nz)) ; CS%u_av(:,:,:) = 0.0 +ALLOC_(CS%v_av(isd:ied,JsdB:JedB,nz)) ; CS%v_av(:,:,:) = 0.0 +ALLOC_(CS%h_av(isd:ied,jsd:jed,nz)) ; CS%h_av(:,:,:) = GV%Angstrom_H +!$omp target enter data map(to: CS%eta, CS%u_av, CS%v_av, CS%h_av) +``` +A second batch (`uhbt`, `visc_rem_u/v`, `pbce`, `eta_PF`, `u_accel_bt/v_accel_bt`) is allocated in +`initialize_dyn_split_RK2` at `:1631-1648`, several with `map(alloc:)` instead of `map(to:)` — a +deliberate choice, flagged by an in-source `TODO`: +```fortran +! MOM_dynamics_split_RK2.F90:1350-1351 +! TODO: Are these initializations necessary? If not, then we can do +! map(alloc:) rather than map(to:) +``` +(i.e. arrays that are always written by a kernel before being read don't need the host-computed +initial zero copied over — `map(alloc:)` skips that copy; arrays read before first write, e.g. `eta` +which may seed itself from `h` on device, need `map(to:)`.) + +Bare `pointer` members exist for restart-registry targeting: `real, pointer, dimension(:,:) :: +taux_bot => NULL()` (`:151`), `tauy_bot` (`:153`), and `type(BT_cont_type), pointer :: BT_cont => +NULL()` (`:155`) (see `02-pointer-usage.md` for why). + +Nested children — **mixed by-value and by-pointer in the same type**: +```fortran +! MOM_dynamics_split_RK2.F90:243-264 +type(hor_visc_CS) :: hor_visc !< by value +type(continuity_CS) :: continuity_CSp !< by value (continuity_CS = continuity_PPM_CS, see §3.3) +type(CoriolisAdv_CS) :: CoriolisAdv !< by value +type(PressureForce_CS) :: PressureForce_CSp !< by value +type(vertvisc_CS), pointer :: vertvisc_CSp => NULL() !< pointer +type(set_visc_CS), pointer :: set_visc_CSp => NULL() !< pointer +type(barotropic_CS) :: barotropic_CSp !< by value +type(SAL_CS) :: SAL_CSp !< by value +type(tidal_forcing_CS) :: tides_CSp !< by value +type(harmonic_analysis_CS) :: HA_CSp !< by value +type(ALE_CS), pointer :: ALE_CSp => NULL() !< pointer +``` +**Only the children that own device-resident arrays are separately entered** onto the device with +their own `map(alloc:)` immediately before their `_init` call, in `initialize_dyn_split_RK2` +(`:1689-1747`) — namely `continuity_CSp`, `PressureForce_CSp`, `hor_visc`, `barotropic_CSp` (all +by-value) and `vertvisc_CSp` (pointer). **Not every by-value child gets its own map:** +`CoriolisAdv` (`:1692`), `SAL_CSp` (`:1695`), `tides_CSp` (`:1696`) and `HA_CSp` (`:1698`) are +`_init`-ed with **no preceding `map(alloc:)`** (verified: `grep` finds no `map(...CS%CoriolisAdv...)` +etc. anywhere in the module) — they ride along inside the parent's whole-struct +`map(alloc: CS%dyn_split_RK2_CSp)` (`MOM.F90:3259`) or are effectively host-only parameter holders. +```fortran +! MOM_dynamics_split_RK2.F90:1689, 1704, 1708, 1711-1712, 1744 (the mapped children) +!$omp target enter data map(alloc: CS%continuity_CSp) +call continuity_init(Time, G, GV, US, param_file, diag, CS%continuity_CSp, CS%OBC) +call CoriolisAdv_init(...) ! :1692 — NO map(alloc: CS%CoriolisAdv) +... +!$omp target enter data map(alloc: CS%PressureForce_CSp) +call PressureForce_init(...) +!$omp target enter data map(alloc: CS%hor_visc) +call hor_visc_init(Time, G, GV, US, param_file, diag, CS%hor_visc, ADp=CS%ADp) +allocate(CS%vertvisc_CSp) +!$omp target enter data map(alloc: CS%vertvisc_CSp) +call vertvisc_init(...) +... +!$omp target enter data map (alloc: CS%barotropic_CSp) +call barotropic_init(...) +``` +This is the "attach a scalar struct first, let its own `_init` attach its array members second" +pattern discussed in §5 — it is the mechanism, not `ALLOCABLE_` vs plain `allocate`, that determines +whether a member ends up device-resident. + +Teardown, `end_dyn_split_RK2` (`:2049-2089`), mirrors this exactly, member-by-member: +```fortran +! MOM_dynamics_split_RK2.F90:2052-2083 +!$omp target exit data map(delete: CS%barotropic_CSp) +call barotropic_end(CS%barotropic_CSp) +call vertvisc_end(CS%vertvisc_CSp) +deallocate(CS%vertvisc_CSp) +call hor_visc_end(CS%hor_visc) +!$omp target exit data map(delete: CS%hor_visc) +... +DEALLOC_(CS%diffu) ; DEALLOC_(CS%diffv) +!$omp target exit data map(delete: CS%diffu, CS%diffv) +... +deallocate(CS) +``` +(the last `deallocate(CS)` deallocates the pointer `MOM_dyn_split_RK2_CS` object itself, called from +`MOM.F90:4733`: `call end_dyn_split_RK2(CS%dyn_split_RK2_CSp)`). + +### 3.3 `continuity_PPM_CS` (aliased `continuity_CS`) (`MOM_continuity_PPM.F90:41-80`) — params only + +```fortran +! MOM_continuity_PPM.F90:41-80 (full type) +type, public :: continuity_PPM_CS ; private + logical :: initialized = .false. + type(diag_ctrl), pointer :: diag + logical :: upwind_1st, monotonic, simple_2nd, aggress_adjust, vol_CFL, better_iter, & + use_visc_rem_max, marginal_faces + real :: tol_eta, tol_vel, CFL_limit_adjust, h_marg_min + integer :: niblock !< The i block size used in array calculations [nondim]. + integer :: njblock !< The j block size used in array calculations [nondim]. + integer :: nkblock !< The k block size used in reconstruction calculations [nondim]. +end type continuity_PPM_CS +``` +Zero array members — nothing to allocate or map. This is the "k-block/tile size" home: `niblock`, +`njblock`, `nkblock` are the CS parameters mentioned in `00-architecture.md` §5, resolved once at +init and read every timestep by `continuity_PPM`/`zonal_mass_flux`/`meridional_mass_flux` to decide +block extents (`if (niblock == 0) niblock = ...`, `:188`). `MOM_continuity.F90:11` aliases the name: +`use MOM_continuity_PPM, only : continuity_CS=>continuity_PPM_CS`. Embedded by value in the parent +(`MOM_dynamics_split_RK2.F90:246`); because it has no arrays it is never itself the target of an +`!$omp target enter data`/`map` directive anywhere in the tree — it's mapped only as part of its +parent's `map(alloc: CS%continuity_CSp)` (a zero-array struct maps almost for free — just its scalar +bytes). + +### 3.4 `hor_visc_CS` (`MOM_hor_visc.F90:43-254`) — mixed macro-allocatable and plain-allocatable arrays + +Roughly 30 logical/real scalar parameters, then two populations of array members: +```fortran +! MOM_hor_visc.F90:141-153 (macro-allocatable, "standard shape" arrays) +real ALLOCABLE_, dimension(NIMEM_,NJMEM_) :: Kh_bg_xx +real ALLOCABLE_, dimension(NIMEM_,NJMEM_) :: Ah_bg_xx +real ALLOCABLE_, dimension(NIMEM_,NJMEM_) :: reduction_xx +! MOM_hor_visc.F90:145,156-161 (plain, bare allocatable — conditionally-present diagnostics/options) +real, allocatable :: Kh_bg_2d(:,:) +real, allocatable :: Kh_Max_xx(:,:), Ah_Max_xx(:,:), Ah_Max_xx_KS(:,:) +real, allocatable :: n1n2_h(:,:), n1n1_m_n2n2_h(:,:) +``` +Both populations are allocated in `hor_visc_init` (`:2867-2966`), e.g. +```fortran +! MOM_hor_visc.F90:2867-2876 (macro form, unconditional, mapped immediately) +ALLOC_(CS%dx2h(isd:ied,jsd:jed)) ; CS%dx2h(:,:) = 0.0 +... +!$omp target enter data map(alloc: CS%dx2h, CS%dy2h, CS%dx2q, CS%dy2q) +!$omp target enter data map(alloc: CS%dx_dyT, CS%dy_dxT, CS%dx_dyBu, CS%dy_dxBu) +! :2883-2884 (plain form, conditional on a runtime flag) +allocate(CS%Kh_Max_xx(Isd:Ied,Jsd:Jed), source=0.0) +allocate(CS%Kh_Max_xy(IsdB:IedB,JsdB:JedB), source=0.0) +``` +**The macro-vs-plain choice does not itself decide device residency** — both populations get mapped +explicitly when a kernel needs them. `CS%Kh_Max_xx` (bare `allocatable`, never touched by +`ALLOCABLE_`) is entered onto device later in the same routine: +```fortran +! MOM_hor_visc.F90:3432-3433 +!$omp target enter data map(to: CS%Kh_max_xx) if (CS%Laplacian) +!$omp target enter data map(to: CS%Kh_max_xy) & +``` +The real distinction: macro-allocatable members are the *unconditional*, "always exists, always +canonical (isd:ied)-shaped" arrays declared inline in the type using the `SZ*`-family conventions; +plain-`allocatable` members are typically **conditionally allocated** (only if a particular +Smagorinsky/Leith/anisotropic/Zanna-Bolton option is on) so `allocated(CS%x)` doubles as both the +memory-presence flag and the physics on/off flag — using `ALLOCABLE_` (which is a no-op attribute +tweak in static mode, meaningless for a conditionally-present array) would not fit that dual-purpose +usage, so these are always genuine plain Fortran `allocatable`, checked with `if (allocated(...))` +at deallocation (`hor_visc_end`, `:3701-3729`: `if (allocated(CS%Kh_Max_xx)) deallocate(CS%Kh_Max_xx)`). + +### 3.5 `vertvisc_CS` (`MOM_vert_friction.F90:56-196`) — mostly scalars plus interface-staggered arrays + +```fortran +! MOM_vert_friction.F90:100-111 +real ALLOCABLE_, dimension(NIMEMB_PTR_,NJMEM_,NK_INTERFACE_) :: a_u !< u-drag coeff at interfaces +real ALLOCABLE_, dimension(NIMEMB_PTR_,NJMEM_,NK_INTERFACE_) :: a_u_gl90 +real ALLOCABLE_, dimension(NIMEMB_PTR_,NJMEM_,NKMEM_) :: h_u !< effective layer thickness +real ALLOCABLE_, dimension(NIMEM_,NJMEMB_PTR_,NK_INTERFACE_) :: a_v, a_v_gl90 +real ALLOCABLE_, dimension(NIMEM_,NJMEMB_PTR_,NKMEM_) :: h_v +real, pointer, dimension(:,:) :: a1_shelf_u => NULL() !< pointer: restart target for ice-shelf coupling +real, pointer, dimension(:,:) :: a1_shelf_v => NULL() +... +type(PointAccel_CS), pointer :: PointAccel_CSp => NULL() !< child CS, pointer +``` +`vertvisc_CS` is itself a **pointer** member of `MOM_dyn_split_RK2_CS` (`vertvisc_CSp`, +`MOM_dynamics_split_RK2.F90:252`), `allocate`d explicitly (`allocate(CS%vertvisc_CSp)`, +`:1711`) rather than being a plain embedded value like `hor_visc`/`continuity_CSp` — because +`vertvisc_init` needs to hand a stable address to `PointAccel`/restart registrations that outlive the +call, and (per `02-pointer-usage.md`) the module wanted an explicit-lifetime object rather than an +implicitly-copied value component. + +### 3.6 Summary: allocatable/pointer choice per CS, parent relationship + +| CS type | Member-in-parent kind | Parent | Array members | Alloc site (parent map) | +|---|---|---|---|---| +| `MOM_control_struct` | plain value (driver-owned) | `MOM_driver.F90:174` | macro (`h,T,S,u,v,...`) | `MOM_driver.F90:282` (`map(alloc: MOM_CSp)`) | +| `MOM_dyn_split_RK2_CS` | `pointer` | `MOM_control_struct` | macro (`CAu,PFu,diffu,...`) | `MOM.F90:3258-3259` | +| `continuity_PPM_CS` | embedded value | `MOM_dyn_split_RK2_CS` | none | `MOM_dynamics_split_RK2.F90:1689` | +| `hor_visc_CS` | embedded value | `MOM_dyn_split_RK2_CS` | macro + plain mix | `MOM_dynamics_split_RK2.F90:1708` | +| `barotropic_CS` | embedded value | `MOM_dyn_split_RK2_CS` | (large, not detailed here) | `MOM_dynamics_split_RK2.F90:1744` | +| `vertvisc_CS` | `pointer` | `MOM_dyn_split_RK2_CS` | macro (interface-staggered) | `MOM_dynamics_split_RK2.F90:1711-1712` | +| `set_visc_CS` | `allocatable` (was embedded value; changed by `81680c15d`/`c82e1254a`) | `MOM_control_struct` | mostly scalar/diag | `MOM.F90:3710` (after `c82e1254a`) | +| `vertvisc_type` (`visc`) | `allocatable` (was embedded value; changed by `81680c15d`) | `MOM_control_struct` | mixed allocatable/pointer, see §4 | `MOM.F90:3277-3278` | + +--- + +## 4. `MOM_variables.F90` — the shared "bag of state" containers + +### 4.1 `thermo_var_ptrs` (`:79-131`) + +```fortran +! MOM_variables.F90:81-98 (excerpt) +real, pointer :: T(:,:,:) => NULL() !< potential temperature — POINTER +real, pointer :: S(:,:,:) => NULL() !< salinity — POINTER +real, pointer :: p_surf(:,:) => NULL() !< POINTER (conditionally allocated) +type(EOS_type), pointer :: eqn_of_state => NULL() +... +real, allocatable, dimension(:,:,:) :: SpV_avg !< genuinely ALLOCATABLE (no external alias needed) +``` +`T`/`S` are pointers purely so `CS%tv%T => CS%T` (`MOM.F90:3119`) can alias the *same* physical memory +as the dycore's macro-allocatable `CS%T`; there is no separate allocation for `tv%T` — it is set once +`CS%T` exists. `p_surf`/`frazil`/`salt_deficit` are allocated conditionally with plain +`allocate(..., source=0.0)` (`MOM.F90:3165-3170`: `if (use_p_surf_in_EOS) allocate(CS%tv%p_surf(isd:ied,jsd:jed), source=0.0)`) +— pointer, not macro-allocatable, precisely because they're conditional and because +`register_restart_field` needs a stable `target`. + +### 4.2 `vertvisc_type` (`:258-313`) — the textbook hybrid, with the exact cited comment + +```fortran +! MOM_variables.F90:259-292 (allocatable drag/BBL fields) +real, allocatable, dimension(:,:) :: & + bbl_thick_u, bbl_thick_v, kv_bbl_u, kv_bbl_v, ustar_BBL, & + BBL_meanKE_loss, BBL_meanKE_loss_sqrtCd, taux_shelf, tauy_shelf +real, allocatable, dimension(:,:,:) :: Ray_u, Ray_v + +! MOM_variables.F90:294-312 (pointer fields, WITH THE EXACT COMMENT) +! The following elements are pointers so they can be used as targets for pointers in the restart registry. +real, pointer, dimension(:,:) :: MLD => NULL() +real, pointer, dimension(:,:) :: h_ML => NULL() +real, pointer, dimension(:,:) :: sfc_buoy_flx => NULL() +real, pointer, dimension(:,:,:) :: Kd_shear => NULL() +real, pointer, dimension(:,:,:) :: Kv_shear => NULL() +real, pointer, dimension(:,:,:) :: Kv_shear_Bu => NULL() +real, pointer, dimension(:,:,:) :: Kv_slow => NULL() +real, pointer, dimension(:,:,:) :: TKE_turb => NULL() +``` +Allocated via `safe_alloc_ptr` (`MOM_safe_alloc.F90:72-94`, a guarded `if (.not.associated(ptr)) +allocate(ptr(...), source=0.0)` helper) in `set_visc_register_restarts` +(`MOM_set_viscosity.F90:2899-2901`: `call safe_alloc_ptr(visc%Kv_shear, isd, ied, jsd, jed, nz+1)`). +Device mapping of `Kv_shear`/`Kv_shear_Bu` is *not* colocated with the allocation — it happens later, +inside `set_visc_init` (after the surrounding `visc` struct itself has been `target update`d), per the +fix in `c82e1254a` (§6.2). + +### 4.3 `accel_diag_ptrs` (`:167-238`) / `cont_diag_ptrs` (`:241-255`) + +Both are **all-pointer** diagnostic-alias structs — no allocation of their own; every member is +pointed at an array owned elsewhere (`Accel_diag%diffu => CS%diffu`, `MOM_dynamics_split_RK2.F90:1664` +etc.). `CS%ADp` in `MOM_control_struct` is `allocatable` (`:340`, the struct itself is allocated, +`allocate(CS%ADp)`, `MOM.F90:3189`) but every field *inside* the allocated struct is a bare pointer +aliasing someone else's macro-allocatable array — the struct exists purely for the diagnostics layer +to hold one handle instead of a dozen. + +### 4.4 `BT_cont_type` (`:317-352`) — all genuinely allocatable + +```fortran +! MOM_variables.F90:318-347 (representative) +real, allocatable :: FA_u_EE(:,:), FA_u_E0(:,:), FA_u_W0(:,:), FA_u_WW(:,:) +real, allocatable :: uBT_WW(:,:), uBT_EE(:,:) +real, allocatable :: h_u(:,:,:), h_v(:,:,:) +type(group_pass_type) :: pass_polarity_BT, pass_FA_uv +``` +No pointers at all — `BT_cont_type` is never a restart-registry target and never aliased by another +module, so there is no forcing reason for pointer semantics; `alloc_BT_cont_type`/ +`dealloc_BT_cont_type` (`MOM_variables.F90:567-639`) do plain `allocate(BT_cont%FA_u_EE(...))`. The +*owning* member, however, is a pointer: `type(BT_cont_type), pointer :: BT_cont => NULL()` +(`MOM_dynamics_split_RK2.F90:155`) — because `BT_cont` is conditionally allocated (only if +`marginal_faces`/certain barotropic options are set) and is handed out to several call sites +(`horizontal_viscosity(..., hu_cont=CS%BT_cont%h_u, ...)`, `:1754`) that need `associated()` gating. + +--- + +## 5. The CS object graph — nesting tree with allocation-site line numbers + +``` +MOM_CSp : type(MOM_control_struct) ! plain value, driver-owned + MOM_driver.F90:174 (declared) + MOM_driver.F90:282 !$omp target enter data map(alloc: MOM_CSp) <- BEFORE initialize_MOM runs +│ +├─ CS%G_in : ocean_grid_type, allocatable MOM.F90:233 (decl) / :2985 allocate / :3034 map(to:) +├─ CS%G : ocean_grid_type, pointer => G_in or rotated copy MOM.F90:234 +├─ CS%GV : verticalGrid_type, pointer MOM_verticalGrid.F90:124 allocate(GV) +├─ CS%tv : thermo_var_ptrs, allocatable MOM.F90:244 (decl) / :2564 allocate(CS%tv) +├─ CS%visc : vertvisc_type, allocatable MOM.F90:261 (decl) / :3277 allocate / :3278 map(alloc:) +├─ CS%ADp : accel_diag_ptrs, allocatable MOM.F90:340 (decl) / :3189 allocate / :3190 map(alloc:) +├─ CS%CDp : cont_diag_ptrs, embedded value MOM.F90:342 +├─ CS%h,T,S,u,v,uh,vh,uhtr,vhtr : macro-allocatable MOM.F90:205-216 (decl) / :3110-3185 ALLOC_+map(to:) +├─ CS%set_visc_CSp : set_visc_CS, allocatable MOM.F90:420 (decl) / :3710ish allocate (post c82e1254a) +└─ CS%dyn_split_RK2_CSp : MOM_dyn_split_RK2_CS, pointer MOM.F90:407 (decl) + │ MOM.F90:3258 allocate / :3259 map(alloc:) + │ + ├─ CS%CAu,PFu,diffu,eta,u_av,h_av,... : macro-allocatable + │ MOM_dynamics_split_RK2.F90:90-147 (decl) + │ MOM_dynamics_split_RK2.F90:1352-1368, 1631-1648 (ALLOC_ + map(to:)/map(alloc:)) + │ + ├─ CS%hor_visc : hor_visc_CS, embedded value + │ MOM_dynamics_split_RK2.F90:244 (decl) / :1708 map(alloc:) / hor_visc_init allocates its own arrays + │ + ├─ CS%continuity_CSp : continuity_CS(=continuity_PPM_CS), embedded value + │ MOM_dynamics_split_RK2.F90:246 (decl) / :1689 map(alloc:) — params only, no arrays + │ + ├─ CS%CoriolisAdv : CoriolisAdv_CS, embedded value + │ MOM_dynamics_split_RK2.F90:248 (decl) + │ + ├─ CS%PressureForce_CSp : PressureForce_CS, embedded value + │ MOM_dynamics_split_RK2.F90:250 (decl) / :1704 map(alloc:) + │ + ├─ CS%barotropic_CSp : barotropic_CS, embedded value + │ MOM_dynamics_split_RK2.F90:256 (decl) / :1744 map(alloc:) + │ + ├─ CS%vertvisc_CSp : vertvisc_CS, pointer + │ MOM_dynamics_split_RK2.F90:252 (decl) / :1711 allocate / :1712 map(alloc:) + │ + └─ CS%set_visc_CSp : set_visc_CS, pointer (this module's own alias, distinct object from CS%set_visc_CSp above) + MOM_dynamics_split_RK2.F90:254 (decl) — set via `CS%set_visc_CSp => set_visc` (:1715), NOT separately allocated +``` + +Teardown mirrors this tree exactly in reverse, member-by-member (`end_dyn_split_RK2`, +`MOM_dynamics_split_RK2.F90:2049-2089`; `MOM_end`, `MOM.F90:4698-4802`), each child's own `_end` routine +called first, then its `!$omp target exit data map(delete: ...)`, then (where allocatable/pointer) +the Fortran `deallocate`. + +--- + +## 6. Three commits that shaped this model + +### 6.1 `81680c15d` — "Allocate MOM CS and both viscosity CS on GPU" + +Root cause: nvfortran was raising ambiguous **"partial presence"** errors on fields of the top-level +`CS` in code paths downstream of `MOM.F90` — i.e. some member of `MOM_control_struct` was present on +device while a sibling member (needed by the same kernel/region) was not, and the compiler could not +reconcile the mixed presence state of a single struct. The fix: change **two** members of +`MOM_control_struct` from plain-embedded-value to `allocatable` (verified against the commit diff — +`git show 81680c15d` touches only these two type declarations, plus `MOM_driver.F90` and +`MOM_set_viscosity.F90`), decoupling their device presence from the parent struct's: +```fortran +! MOM.F90 diff (81680c15d) +- type(ocean_grid_type) :: G_in !< Input grid metric ++ type(ocean_grid_type), allocatable :: G_in !< Input grid metric +... +- type(vertvisc_type) :: visc !< structure containing vertical viscosities, ... ++ type(vertvisc_type), allocatable :: visc !< ... +``` +(`set_visc_CSp` was **not** made `allocatable` here — that came one commit later in `c82e1254a`, +§6.2; this commit only *maps* `set_visc_CSp` onto the device, it does not change its declaration.) + +> **Resolved (2026-07-14):** "Partial presence" is the literal NVIDIA runtime diagnostic, not a +> reconstruction. The NVHPC OpenMP/OpenACC runtime raises a FATAL "partially present" error when a +> mapping's address range partially overlaps an existing present-table entry — exactly the +> whole-struct-over-attached-member overlap described here. Rely on this mechanism as a general rule. +plus explicit `allocate(CS%G_in)` (before `G_in => CS%G_in`) and `allocate(CS%visc)` before their +respective `map(to:)`/`map(alloc:)` directives, and a whole-struct `!$omp target update to(CS)` right +after the grid upload (`MOM.F90:3104`, new in this commit) — which the commit message flags as breaking the project's +own derived-type handling rule ("we do an update(CS) after the grid has been uploaded... this needs +exploration") but was needed and apparently harmless once `G_in`/`visc` were decoupled as allocatables. +The commit further notes the `G_in`→allocatable change specifically fixed a **performance** problem +("excessive grid transfers... severely degrading performance"), distinct from the correctness +"partial presence" issue that motivated `visc`. + +### 6.2 `c82e1254a` — "vertvisc: Fix CS memory management" + +A regression from `81680c15d`: `CS%visc` was being **allocated twice** — once implicitly by giving it +the `allocatable` attribute plus an early `allocate(CS%visc)`, and (unclear from the single commit, +but per the message) a second time overwriting the pointers `visc%Kv_shear`/`visc%Kv_shear_Bu` that +`set_visc_register_restarts` had already set up, corrupting `associated()` state used for flow control +in device kernels — causing addressing errors in `double_gyre` runs. Fix, concretely: +- `CS%set_visc_CSp` changed from embedded value to `allocatable` (`MOM.F90` diff: + `- type(set_visc_CS) :: set_visc_CSp` / `+ type(set_visc_CS), allocatable :: set_visc_CSp`), + matching the same pattern used for `G_in`/`visc` in the prior commit ("This may not be needed, but + it is consistent with other types"). +- The single combined `!$omp target enter data map(to: CS%visc, CS%set_visc_CSp)` before + `set_visc_init` was split: `CS%visc` is now updated with `!$omp target update to(visc)` **inside** + `set_visc_init` (`MOM_set_viscosity.F90:3322-3324`), and `CS%set_visc_CSp` gets its own + `allocate(CS%set_visc_CSp)` + `map(alloc:)` in `MOM.F90` immediately before the call. +- The device `map(alloc:)`/`map(to:)` for `visc%Kv_shear`/`visc%Kv_shear_Bu` was **moved out of** + `set_visc_register_restarts` (removed the early `!$omp target enter data map(alloc: visc%Kv_shear)` + right after `safe_alloc_ptr`) and **into** `set_visc_init`, applied only *after* the `target update + to(visc)` scalar sync: + ```fortran + ! MOM_set_viscosity.F90:3322-3330 (post-fix) + !$omp target update to(visc) + !$omp target update to(CS) + !$omp target enter data map(to: visc%Kv_shear) if (associated(visc%Kv_shear)) + !$omp target enter data map(to: visc%Kv_shear_Bu) if (associated(visc%Kv_shear_Bu)) + ``` +- In `MOM_vert_friction.F90`'s `vertvisc_coef`, the stale in-source comment explaining that + `Kv_shear` is "persistently mapped... so map(to:) would not copy host updates" was removed along + with the workaround it justified; `Kv_shear_Bu` now uses `!$omp target update to(visc%Kv_shear_Bu)` + (was: a redundant `map(to:)` re-enter-data every call) matching how `Kv_shear` was already handled, + and the matching `map(release: visc%Kv_shear_Bu)` at exit was deleted (no longer needed since it's + not separately entered per-call). + +This pair of commits establishes the working rule for this codebase (visible nowhere as a written +rule except in these two commit messages): **any CS-type field that participates in cross-call +`associated()`-gated flow control, or whose presence must be decoupled from a sibling field's +presence, should be declared `allocatable` (not an embedded value) in its parent**, so its device +lifetime can be managed independently with its own `map(alloc:)`/`allocate()` pair rather than being +swept up in whatever bulk-struct directive covers its parent. + +### 6.3 `1865612de` — "Convert structs of arrays to flat arrays" + +Scope: `src/tracer/MOM_tracer_hor_diff.F90` only (`tracer_epipycnal_ML_diff`, `:757` region). Before: +```fortran +! pre-commit (struct-of-arrays, one small array per j-index) +type(p2d), dimension(SZJ_(G)) :: deep_wt_Lu, deep_wt_Ru, hP_Lu, hP_Ru +... +do j=js,je + k_size = max(2*max_srt(j),1) + allocate(deep_wt_Lu(j)%p(IsdB:IedB,k_size)) + ... + !$omp target enter data map(alloc: deep_wt_Lu(J)%p, deep_wt_Ru(J)%p, hP_Lu(J)%p, hP_Ru(J)%p, & + !$omp k0a_Lu(j)%p, k0a_Ru(j)%p, k0b_Lu(j)%p, k0b_Ru(j)%p) +enddo +``` +After: a single flat `real, dimension(:,:,:), allocatable :: deep_wt_Lu, ...` sized once +(`k_size = max over all j of 2*max_srt(j)`, via a `do concurrent ... DO_LOCALITY(reduce(max:k_size))` +reduction) and allocated/entered **once**: +```fortran +allocate(k0a_Lu(IsdB:iedB,k_size,jsd:jed)) +... +!$omp target enter data map(alloc: deep_wt_Lu, deep_wt_Ru, hP_Lu, hP_Ru, k0a_Lu, k0a_Ru, k0b_Lu, k0b_Ru) +``` +The commit message is explicit about the mechanism: each `type(p2d)`/`type(p2di)` element was a +**separate derived-type instance with its own pointer-array component**, so mapping `SZJ_(G)` +(`O(njglobal)`) of them onto the device meant `O(njglobal)` individual "attach the array component to +this struct instance" operations — each one a distinct OpenMP mapping bookkeeping entry — rather than +one attach of one flat array. "Flattening these arrays halves time when compiling for GPU. Lots of +time was being spent 'attaching' and 'detaching' the member arrays to/from each struct on the GPU." +Tradeoff acknowledged in the message: ~20% more memory (padding every j-slice out to the same +`k_size`, versus each j's exact `2*max_srt(j)`), traded for the attach/detach overhead reduction. This +is the same underlying cost (`00-architecture.md` calls it "the offload unit is the whole CS object +with its allocatable members... deep-copy of derived-type member arrays is expensive") observed at a +finer grain — even without a big CS, an array *of small derived-type instances each holding an array* +pays the same per-instance attach tax, and the fix generalizes: prefer one flat array with an extra +dimension over an array of one-member-array structs whenever the outer array's extent is a loop-index +range rather than a true type/kind distinction. + +--- + +## 7. Practical checklist derived from the above + +1. Deciding embedded-value vs `allocatable` vs `pointer` for a new child CS field: + - No arrays, always exists → embedded value is fine (`continuity_PPM_CS` pattern, §3.3). + - Has arrays, always exists for the life of the parent, no `associated()` gating needed elsewhere + → embedded value still works (`hor_visc`, `barotropic_CSp`), mapped via the parent's + `map(alloc:)` + its own `_init`'s per-array `map(to:)`/`map(alloc:)`. + - Conditionally allocated, or needs independent device-presence lifetime from its parent, or is a + restart-registry target → `allocatable` (post-`81680c15d`/`c82e1254a` idiom) or `pointer` (older + idiom, still used for `vertvisc_CSp`/`set_visc_CSp` *inside* `MOM_dyn_split_RK2_CS`, as opposed + to the top-level `MOM_control_struct` copies which are now `allocatable`). +2. Every macro-allocatable array needs its own `ALLOC_`/zero-init/`!$omp target enter data + map(to:|alloc:)` triplet at init and matching `DEALLOC_`/`map(delete:)` at `_end` — see + `00-architecture.md` §9's "Map a new CS array" recipe; this document's §3 gives the concrete + line-referenced exemplars to copy from. +3. Avoid arrays of small derived-type instances that each carry an array component (the `type(p2d), + dimension(SZJ_(G))` anti-pattern) when the outer index is a loop range, not a real type + distinction — flatten to one array with an extra dimension (`1865612de`). +4. Symmetric-memory offsets are a runtime `ALLOC_`-call-site concern in dynamic mode, not a macro + concern — always compute B-point bounds from `hor_index_type`/`ocean_grid_type`'s `IsdB`/`JsdB` + etc. (which already encode the `-1` symmetric offset), never hand-roll `isd-1`. + +### 7.1 Prescriptive rules for adding GPU-resident CS state (grounded in cited code) + +These are the load-bearing rules distilled from §§3–6; follow them when introducing a new CS member +or a new child CS on `dev/gpu`. + +- **Declare persistent 3-D/2-D work arrays as macro-allocatable, not `pointer`.** Use + `real ALLOCABLE_, dimension(SZ*-family macros) :: x` (e.g. `MOM_dynamics_split_RK2.F90:90`, + `MOM.F90:205`). Reserve `pointer` **only** for arrays that must be a `target` in the restart + registry or aliased across modules (the `MOM_variables.F90:294` comment — *"pointers so they can be + used as targets for pointers in the restart registry"* — is the canonical justification; e.g. + `visc%Kv_shear`, `taux_bot`). Pointers complicate device mapping and force `associated()`-gated + paths; do not reach for them by default. +- **Declare a new *child CS* as `allocatable`, not embedded-value, unless it is trivially small and + always-present.** Embedded-value is acceptable only for the zero-array / small-scalar case + (`continuity_PPM_CS`, §3.3) or an always-present array-owning child whose presence never needs to be + decoupled from a sibling's (`hor_visc`, `barotropic_CSp`). The moment a child is **conditionally + allocated**, needs an **independent device-presence lifetime**, or participates in cross-call + **`associated()`-gated flow control**, make it `allocatable` — this is the explicit lesson of + `81680c15d` (`G_in`, `visc`) and `c82e1254a` (`set_visc_CSp`). A top-level embedded-value struct is + an anti-pattern precisely because its device presence gets entangled with the parent's (the + "partial presence" failure, §6.1). +- **Order at init: allocate/attach the parent *shell* before its members.** The parent CS is entered + with `map(alloc:)` (scalars only) *before* its `_init` runs, and each array member is entered by its + own `_init` afterwards. Concretely: `MOM_CSp` is `map(alloc:)`-ed in the driver + (`MOM_driver.F90:282`) **before** `initialize_MOM` (`:286`); `CS%dyn_split_RK2_CSp` is + `allocate`+`map(alloc:)`-ed (`MOM.F90:3258-3259`) **before** its arrays are allocated in + `register_restarts_dyn_split_RK2`/`initialize_dyn_split_RK2`. Never map a member array before the + struct that contains it exists on device. +- **Co-locate the map with the allocate, and mirror it in `_end`.** Each macro-allocatable array gets + an `ALLOC_(...)` + zero-init + `!$omp target enter data map(to:|alloc:)` triplet in `*_init` + (`MOM_dynamics_split_RK2.F90:1352-1368`) and a matching `DEALLOC_(...)` + `!$omp target exit data + map(delete: ...)` in `*_end` (`:2065-2083`), member-by-member in reverse order. Choose `map(to:)` + only when the host-computed initial value is read before the first device write; otherwise + `map(alloc:)` skips the copy (the in-source `TODO` at `:1350-1351` documents exactly this choice). + **Exception (the `c82e1254a` rule):** for a `pointer` array whose `associated()` state is set up in a + *separate* registration routine (`safe_alloc_ptr`), do **not** map it at the allocation site — map it + later, *after* a `!$omp target update to()`, guarded by `if (associated(...))` + (`MOM_set_viscosity.F90:3323-3330`). Mapping it early double-allocates and corrupts the pointer state. +- **Never build an array of small derived types that each own an array component when the outer index + is a loop range** (`type(p2d), dimension(SZJ_(G))`, §6.3) — every instance pays a separate device + attach/detach. Flatten to one `allocatable` array with an extra dimension (`1865612de`). This is the + same "deep-copy of derived-type member arrays is expensive" cost the whole-CS model pays, seen at + finer grain. +- **`GV` is a genuine exception to "map every array."** Only `GV`, `GV%Rlay`, `GV%g_prime` are mapped + (`MOM.F90:3650`), and only after rescaling; the rest of the vertical grid reaches kernels by-value. + Do not assume vertical-grid arrays are device-resident by default (see the §2 correction). + +--- + +## Verification notes + +Opus verification pass (source + git only; no build/run). Every line number and macro expansion in +this document was checked against the current `dev/gpu` tree unless flagged below. + +**Confirmed against code/git:** +- All `MOM_memory_macros.h` expansions and line ranges: attribute macros (`:101-110`), heap-shape + macros (`:114-161`), the `NIMEMB_PTR_`→`:` (dynamic, `:122,125`) vs `NIMEMB_PTR_`→`NIMEMB_` (static, + `:53,56`) subtlety, `SZ*` family (`:167-182`), `SZD*` (`:189-195`). The document's central claim — + that dynamic-mode symmetric offsets live in the runtime `ALLOC_` bounds (`IsdB`/`JsdB`), not the type + declaration, while static mode bakes them into the macro — is correct. +- `config_src/memory/{dynamic_symmetric,dynamic_nonsymmetric}/MOM_memory.h`: byte-identical except the + `SYMMETRIC_MEMORY_` `#define`/`#undef` at `:37`; both `#undef STATIC_MEMORY_` (`:41`), `NIHALO_=2`. +- `hor_index_init` B-bound logic (`MOM_hor_index.F90:89-99`), `MOM_grid_init` mirror (`:303-313`), + `allocate_metrics` ALLOC_ sites (`:547-548`). +- `MOM_control_struct` layout and every allocation/map site in `MOM.F90`: `G_in` (`:233`/`:2985`/`:3034`), + `visc` (`:261`/`:3277-3278`), `tv` (`:244`/`:2564`), `ADp` (`:340`/`:3189-3190`), + `dyn_split_RK2_CSp` (`:407`/`:3258-3259`), `set_visc_CSp` (`:420`/`:3709-3711`), prognostic state + (`:205-216`/`:3110-3115`), `tv%T => CS%T` (`:3119`), `p_surf` (`:3165`), `map(alloc: MOM_CSp)` before + `initialize_MOM` (`MOM_driver.F90:282` before `:286`). +- `MOM_dyn_split_RK2_CS` full member list and allocation/teardown (`:89-281`, `:1352-1368`, + `:1631-1648`, `:1689-1747`, `:2049-2089`, module doc `:2113-2116`). +- `continuity_PPM_CS` (`:41-80`, zero arrays), `vertvisc_CS` arrays (`:100-111`), `verticalGrid_type` + arrays and `verticalGridInit`/`End` (`:106-247`/`:361-368`). +- `MOM_variables.F90`: `thermo_var_ptrs` (`:79`), `vertvisc_type` with the exact restart-registry + comment at `:294`, `accel_diag_ptrs` (`:167`), `cont_diag_ptrs` (`:241`), `BT_cont_type` (`:317`), + `safe_alloc_ptr` (`MOM_safe_alloc.F90`, `if (.not.associated(ptr))` guard), `set_visc` post-fix + block (`MOM_set_viscosity.F90:3322-3330`). +- All three commit narratives (`81680c15d`, `c82e1254a`, `1865612de`) match their commit messages and + diffs, including the "partial presence" motivation, the double-allocation/`associated()`-corruption + regression fix, and the `~700k→~830k (~20%)` memory tradeoff. `1865612de` scope confirmed as + `MOM_tracer_hor_diff.F90` only. + +**Corrected:** +1. **§2 (major):** the claim that no `GV` member is ever mapped and "it never appears in an `!$omp + target enter data` list" is **false**. `MOM.F90:3650` has an active `map(to: GV, GV%Rlay, + GV%g_prime)`, with an instructive commented-out earlier attempt at `:3057`. Rewritten with the + correct facts. +2. **§3.2:** the blanket claim that *"each by-value child is separately entered with its own + `map(alloc:)`"* overstated the code — `CoriolisAdv`, `SAL_CSp`, `tides_CSp`, `HA_CSp` are by-value + children with **no** preceding `map(alloc:)` (`grep`-verified). Only `continuity_CSp`, + `PressureForce_CSp`, `hor_visc`, `barotropic_CSp` (+ pointer `vertvisc_CSp`) are individually mapped. +3. **§6.1:** "change **three** members … to `allocatable`" corrected to **two** (`G_in`, `visc`); the + `git show 81680c15d` diff changes only those two declarations. `set_visc_CSp` became `allocatable` + in `c82e1254a`, not here. +4. **Minor line numbers:** `MOM_end` subroutine starts at `:4698` (was cited as `:4680`); the three + `DEALLOC_(CS%u/v/h)` are all on `:4778` (was `:4778-4779`); `NK_INTERFACE_` reclassified in the + table from "dummy-arg form" to a heap-shape macro (`NK_+1` in static mode). + +**Confidence:** High. Nearly all line numbers and macro/expansion details were verified exact against +the working tree; the few discrepancies were small offsets (now fixed) plus the two substantive +overstatements above (GV mapping, by-value-child mapping) and one miscount (two vs three members). +The subtle symmetric-memory / `ALLOC_`-site vs macro-expansion story — the document's core technical +thesis — is fully correct. diff --git a/knowledge/gpu-knowledge/02-pointer-usage.md b/knowledge/gpu-knowledge/02-pointer-usage.md new file mode 100644 index 0000000..79ec89c --- /dev/null +++ b/knowledge/gpu-knowledge/02-pointer-usage.md @@ -0,0 +1,471 @@ +# Pointer Usage Across MOM6 (dev/gpu) — Aliasing Hazards for GPU Offload + +> Companion to `00-architecture.md` §2 (CS pattern) and §3 (memory conventions). This document +> inventories every recurring `pointer` idiom in the tree, ties each to a concrete device-mapping +> consequence, and reconstructs the multi-GPU/pointer bugs that have already been hit and fixed on +> `dev/gpu`. Read `00-architecture.md` first. + +--- + +## 1. Why pointers exist at all in this codebase + +`grep -rn "pointer" src/ --include=*.F90` matches in essentially every module; the busiest files are +`MOM.F90` (108), `MOM_diag_mediator.F90` (107), `MOM_open_boundary.F90` (89), `MOM_restart.F90` (62), +`MOM_variables.F90` (54), `MOM_dynamics_split_RK2.F90` (53). Three genuinely distinct motives account +for nearly all of them; a fourth is incidental and should be removed opportunistically. + +1. **Restart-registry targets (unavoidable, by construction of `MOM_restart.F90`).** Every + `register_restart_field_ptr*d` entry point takes the field as a `target, intent(in)` dummy and + stores a bare Fortran pointer alias to it: + ```fortran + ! src/framework/MOM_restart.F90:207-239 (register_restart_field_ptr3d) + real, dimension(:,:,:), target, intent(in) :: f_ptr + ... + CS%var_ptr3d(CS%novars)%p => f_ptr + ``` + `MOM_restart_CS` (`:124-134`) holds `type(p0d/p1d/p2d/p3d/p4d), pointer :: var_ptrNd(:)` arrays — + heterogeneous, dynamically-sized collections of aliases to arbitrary host arrays scattered across + many CSs, so they **must** be pointers (or `target` actuals bound at call time); there is no + allocatable equivalent that can alias a *pre-existing* array owned by someone else. This is the + one class of pointer that cannot be designed away without redesigning the restart mechanism + itself. + - Because the registry only needs a *host*-side alias for read/write I/O, restart-registered + fields do not need special device treatment purely on that account — but any field that is + *conditionally* allocated (only when a physics option is on) is forced to be declared + `pointer` in its owning derived type so `associated()` can gate both the registration call and + later device mapping (see §5). + +2. **Cross-module aliasing of shared "bag of state" containers (`MOM_variables.F90`).** Several + derived types exist purely to let unrelated modules see the same prognostic/diagnostic arrays + without copying: + - `thermo_var_ptrs` (`MOM_variables.F90:79`) — `T`, `S`, `p_surf` are pointers; `tv%T => CS%T` and + `tv%S => CS%S` are set once in `MOM.F90:3119` (`CS%tv%T => CS%T ; CS%tv%S => CS%S`, repeated at + `MOM.F90:3432-3433` after a possible reallocation) so that any routine holding only `tv` (not + `CS`) — the whole EOS/diabatic/ALE call chain — reads/writes the *same* memory as the dycore's + `CS%T`/`CS%S`. + - `ocean_internal_state` (`MOM_variables.F90:138`) — **every** member is `pointer`, aliasing + `T,S,u,v,h,uh,vh,CAu,CAv,PFu,PFv,diffu,diffv,pbce,u_accel_bt,v_accel_bt,u_av,v_av,u_prev,v_prev`. + Populated in one shot in `MOM.F90:3195-3212`: + ```fortran + MOM_internal_state%u => CS%u ; MOM_internal_state%v => CS%v + MOM_internal_state%h => CS%h + MOM_internal_state%uh => CS%uh ; MOM_internal_state%vh => CS%vh + ... + CS%CDp%uh => CS%uh ; CS%CDp%vh => CS%vh + ``` + It exists solely so a single "give me everything for diagnostics" structure can be handed to + ensemble/diagnostic code without copying 3D prognostic arrays. + - `accel_diag_ptrs` (`:167`) / `cont_diag_ptrs` (`:241`) — **all-pointer** diagnostic alias + structs (`CS%ADp`, `CS%CDp`) that let `MOM_vert_friction.F90`, `MOM_CoriolisAdv.F90`, + `MOM_PressureForce_FV.F90`, `MOM_continuity_PPM.F90` each write into a shared accumulator array + for later energy-budget diagnostics, without every producer routine owning the array itself. + +3. **Optional/feature-gated fields (`associated()` as a runtime "is this feature on" flag).** + `vertvisc_type` (`MOM_variables.F90:258`) is explicitly **hybrid**: drag fields + (`bbl_thick_u`, `kv_bbl_u`, `Ray_u`, …) are plain `allocatable` because they always exist, but + `MLD`, `h_ML`, `sfc_buoy_flx`, `Kd_shear`, `Kv_shear`, `Kv_shear_Bu`, `Kv_slow`, `TKE_turb` are + `pointer` with an explicit comment: + ```fortran + ! The following elements are pointers so they can be used as targets for pointers in the + ! restart registry. + real, pointer, dimension(:,:) :: MLD => NULL() + ... + real, pointer, dimension(:,:,:) :: Kd_shear => NULL() + ``` + These are allocated only if the relevant parameterization (KPP/ePBL/CVMix shear/`RiNo_mix`) is + selected at init (`MOM_set_viscosity.F90:2899-2909`, `register_restart_field(visc%Kv_shear, ...)` + guarded by `if (useKPP .or. useEPBL .or. use_CVMix_shear .or. ...)`). Downstream code (e.g. + `vertvisc_coef` in `MOM_vert_friction.F90`) then must use `associated(visc%Kv_shear)` both as a + host feature-flag *and* as an OpenMP `if()` map guard — see §3. + +4. **Incidental pointers that are not load-bearing for aliasing.** Many `local pointer => CS%member` + assignments exist purely as a **naming convenience** so a long subroutine can write `u`,`v`,`h` + instead of `CS%u`, `CS%v`, `CS%h` — they alias, are never reassigned, and could be plain + dummy-argument passing (or associate blocks) instead: + - `MOM.F90:629` (`step_MOM`): `u => CS%u ; v => CS%v ; h => CS%h` (verified). + - `MOM.F90:1272` (`step_MOM_dynamics`): same pattern (verified). + - `MOM_dynamics_split_RK2.F90:426` (`step_MOM_dyn_split_RK2`): + `u_av => CS%u_av ; v_av => CS%v_av ; h_av => CS%h_av ; eta => CS%eta`. These four are the true + incidental aliases in the dycore; the `eta` pointer even carries an explicit source comment + "*This pointer is just used as shorthand for `CS%eta`*" (declared `:385-395`, bound `:426`). + **Correction:** `u,v,h` themselves do *not* appear as local `=> CS%…` aliases in this routine — + they enter as dummy arguments (`u_inst`, `v_inst`, `h`; the velocities carry a `target` + attribute at `:309/:311` because `btstep`/`u_ptr` build pointer views of them). The earlier draft's + "`u,v,h => NULL()` declared at `:151-153`, bound at `:165`" was wrong: `:151-153` are *CS-type + members* (`taux_bot`, `tauy_bot`), and no `u => CS%u` binding exists anywhere in this file. + These are the ones the port doc calls "incidental": they don't gate any `associated()` logic and + aren't restart targets themselves (the *targets*, `CS%u_av`/`CS%eta`/…, are macro-allocatables, not + pointers). They complicate reading the code and (in principle) complicate alias analysis for a + compiler trying to prove no-aliasing for `do concurrent`, but they carry no unique semantic + weight and are candidates for straightforward removal once a porting pass touches these routines. + + **Contrast — reassigned aliases are *not* incidental.** In the same routine, + `u_ptr, v_ptr, uh_ptr, vh_ptr` (declared `:395-400`, comment "*used to alter which fields are + passed to `btstep` with various options*") and the local `p_surf` (§3) are pointers whose **target + changes at runtime** depending on options — they select *which* array `btstep` operates on. These + carry real semantic weight: their `associated()` status and device attachment must be + re-established every call (the map guard cannot be hoisted, see §3's `p_surf` case). Do not lump + them in with the removable `u_av => CS%u_av` shorthand. + +**Rule of thumb going forward:** if a pointer's only job is "give this array a short local name," it +is incidental and removable. If `associated()` on it changes control flow (feature gating) or it +appears as `target` in a `register_restart_field` call, it is load-bearing and must stay a pointer +(or an allocatable held by `target`, see §4). + +--- + +## 2. `tracer_type` / `tracer_registry_type` — an array-of-structs-of-pointers + +`src/tracer/MOM_tracer_types.F90`: +- `tracer_type` (`:13`) has **~20 pointer members** — `t` (the tracer concentration, `:15`), plus + advective/diffusive diagnostic arrays `ad_x, ad_y, ad2d_x, ad2d_y, df_x, df_y, hbd_dfx, hbd_dfy, + advection_xy, t_prev, …` — all `real, dimension(...), pointer => NULL()`. Most of these are + optionally allocated depending on which diagnostics are requested (`associated()` gates their use + throughout `MOM_tracer_advect.F90` and `MOM_tracer_hor_diff.F90`, e.g. `if (associated(Reg%Tr(m)%ad_x))` + at `MOM_tracer_advect.F90:244,777`). +- `tracer_registry_type` (`:129`): `type(tracer_type) :: Tr(MAX_FIELDS_)` — a **fixed-size array of + structs, each stuffed with pointers** (`MAX_FIELDS_ = 50`, `config_src/memory/*/MOM_memory.h:26`). + This is the single densest pointer-in-derived-type-array structure in the tree and is the direct + cause of the bug in §3. + +This AoS-of-pointers pattern recurred as a **performance** problem too, independent of correctness: +commit `1865612de` ("Convert structs of arrays to flat arrays", `src/tracer/MOM_tracer_hor_diff.F90`) +replaced `type(p2d), dimension(SZJ_(G)) :: deep_wt_Lu, hP_Lu, ...` (one pointer-to-2D-array *per j-row*, +each individually `!$omp target enter data`'d — `allocate(deep_wt_Lu(j)%p(...)); !$omp target enter +data map(alloc: deep_wt_Lu(J)%p, ...)` inside a `do j=js,je` loop) with flat +`real, dimension(:,:,:), allocatable :: deep_wt_Lu` arrays mapped once. Commit message: *"Lots of +time was being spent 'attaching' and 'detaching' the member arrays to/from each struct on the +GPU"* — flattening **halved GPU compile/attach time** at the cost of ~20% more memory (700k → 830k +elements allocated per array in the benchmark case). This generalizes §2.3 of `00-architecture.md`: +**any array of pointer-bearing structs is a per-element attach/detach tax; flatten it if it's on a +hot path.** + +--- + +## 3. Aliasing patterns and their `associated()` map guards + +The dominant defensive idiom on `dev/gpu` is: pointer members that may or may not be allocated are +mapped/updated with an `if (associated(...))` clause so the OpenMP directive is a no-op when the +feature is off, instead of erroring on a null/unassociated pointer: + +- `MOM_dynamics_split_RK2.F90:472`: `!$omp target enter data map(to: p_surf) if (associated(p_surf))` + and the matching `MOM_dynamics_split_RK2.F90:939`: `!$omp target exit data map(delete: p_surf) if + (associated(p_surf))`. `p_surf` here is a **local pointer**, resolved just above (`:461-467`) to + either `p_surf_end` (dynamic surface pressure) or `forces%p_surf`, i.e. the alias target changes + every call — the map guard has to be re-evaluated every call because `associated()` can flip. +- `MOM_vert_friction.F90` (`vertvisc_coef`): `!$omp target update to(visc%Kv_shear) if + (associated(visc%Kv_shear))` and `!$omp target update to(visc%Kv_shear_Bu) if + (associated(visc%Kv_shear_Bu))` — see the full history in §5, this exact pair was buggy twice. +- `MOM_set_viscosity.F90` (`set_visc_init`): `!$omp target enter data map(to: visc%Kv_shear) if + (associated(visc%Kv_shear))` / `... map(to: visc%Kv_shear_Bu) if (associated(visc%Kv_shear_Bu))` + — the *allocating* side's mirror of the same guard. +- Throughout `MOM_tracer_advect.F90`: `if (associated(Reg%Tr(m)%ad_x))`, + `if (associated(Reg%Tr(m)%ad_y))`, `if (associated(Reg%Tr(m)%advection_xy))`, + `if (associated(Reg%Tr(m)%ad2d_x))`, `if (associated(Reg%Tr(m)%ad2d_y))` (`:244-266`) gate + device-side zeroing of optional diagnostic accumulators inside `do concurrent`. + +**The pattern to internalize:** `associated()` is not just a host-side null check here — it is a +*device control-flow condition* embedded directly in OpenMP `map`/`update` clauses and inside +`do concurrent` bodies. For it to give correct answers on device, the pointer's association status +(and, once associated, its target's contents) must be **faithfully mirrored to device** — which is +exactly what `map(alloc:)` does *not* do (next section). + +**`allocated()` vs `associated()` — match the intrinsic to the member kind.** `vertvisc_type` is +*hybrid* (§1.3): its always-present drag fields are `allocatable`, its feature-gated fields are +`pointer`. The guard intrinsic differs accordingly and the two are not interchangeable: +`allocated()` for allocatable members, `associated()` for pointer members. Both are used as device +map/update guards side by side — e.g. `MOM.F90:1807-1812` guards the allocatable drag fields with +`!$omp target update from(CS%visc%Ray_u) if (allocated(CS%visc%Ray_u))`, +`... bbl_thick_u) if (allocated(...))`, while the pointer fields (`Kv_shear`, `Kv_shear_Bu`) are +guarded with `if (associated(...))` (above). **Porting rule:** before writing a map guard, check the +member's declaration — an `if (associated(x))` on an `allocatable` array (or vice-versa) is a +compile error at best and silently wrong control flow at worst. + +--- + +## 4. `map(alloc:)` vs `map(to:)` on pointer/derived-type members — the `Reg%Tr(:)` bug + +Two same-day commits by Edward Yang fixed distinct multi-GPU answer-change bugs in +`advect_tracer` (`src/tracer/MOM_tracer_advect.F90`); both are cited together in the branch history +but have **different root causes** and are worth keeping separate. + +### 4a. `e182de310` — "advect_tracer: fix multi gpu answer change" (race on a shared array element) + +Not actually a pointer-aliasing bug, but the immediate predecessor in the same file and commonly +conflated with 4b. The reduction variable `domore_k(k)` was being written from inside two separate +`do concurrent` loops as a plain assignment: +```fortran +domore_k(k) = 0 +do concurrent (j=jsv:jev, domore_u(j,k)) + domore_k(k) = 1 ! every iteration in the do-concurrent writes the SAME element +enddo +``` +This is a write-write race on a single shared array element with no reduction semantics — undefined +under `do concurrent` and answer-dependent on team/thread scheduling, hence non-reproducible +**across GPUs** (and, more subtly, across different launch configurations on the same GPU). The fix +introduces a scalar temporary and an explicit reduction clause: +```fortran +domore_k_tmp = 0 +do concurrent (j=jsv:jev, domore_u(j,k)) DO_LOCALITY(reduce(max:domore_k_tmp)) + domore_k_tmp = 1 +enddo +... +domore_k(k) = domore_k_tmp ! single, well-defined write after the reduction completes +``` +The commit note explains the scalar temporary is required because "`do concurrent` can't use array +elems yet" as the reduction variable. Lesson for porting: **never let a `do concurrent` body write an +indexed array element as an implicit reduction target — always reduce into a scalar first.** + +### 4b. `a774eb331` — "tracer_advect: fix map of Reg%Tr(:)" (the actual pointer/AoS mapping bug) + +```diff +- !$omp target enter data map(to: OBC) map(alloc: domore_u, domore_v, uhr, vhr, uh_neglect, & +- !$omp vh_neglect, hprev, local_advect_scheme, Reg, Reg%Tr(:)) ++ !$omp target enter data map(to: OBC, Reg, Reg%Tr(:)) map(alloc: domore_u, domore_v, uhr, vhr, & ++ !$omp uh_neglect, vh_neglect, hprev, local_advect_scheme) +``` +(and the matching `exit data`: `map(release: hprev, ...)` no longer separately `map(from: hprev)`). + +**Mechanism.** `Reg%Tr(:)` is `tracer_registry_type%Tr(MAX_FIELDS_)` — an array of `tracer_type`, +each element carrying ~20 `pointer` members (§2) plus plain scalars (`advect_scheme`, `ntr`-derived +metadata, diagnostic IDs). The routine immediately (and throughout) tests +`associated(Reg%Tr(m)%ad_x)`, `associated(Reg%Tr(m)%ad_y)`, `associated(Reg%Tr(m)%advection_xy)`, +etc. on device (`:244-266`). `map(alloc: Reg, Reg%Tr(:))` only **reserves device storage** for the +struct array — it does *not* copy the host struct's bytes over, so every scalar and every pointer +descriptor inside `Reg%Tr(m)` starts as **whatever bit pattern the device allocator happened to hand +back** (uninitialized device memory). `associated()` reads that pointer descriptor; with garbage +bits the answer is unpredictable — sometimes true, sometimes false, and the pattern differs by GPU +architecture, driver, and allocator state, i.e. exactly a "multi-GPU answer change": the same source +produces different results depending on which device (or even which run) executes it, because it's +reading uninitialized memory to decide whether to zero a diagnostic array. `map(to: Reg, Reg%Tr(:))` +instead performs the copy: the host's true association status and (for later individually-attached +members like `Reg%Tr(m)%t`, which is separately mapped at `:237` `map(to: Reg%Tr(m)%t)`) correct +target linkage are mirrored to device before any kernel reads them. + +**Generalization — the porting rule:** for a derived type (or array of derived types) that is later +read with `associated()` on device, or whose scalar members feed device control flow, **you must +`map(to:)` it, never `map(alloc:)`.** `map(alloc:)` is only safe for arrays whose entire initial +content is written by device code before being read (pure "workspace" arrays like `domore_u`, `uhr`, +`hprev` in the same directive, which is exactly why those remain `map(alloc:)` in the same line). +Mapping a `pointer`-bearing struct with `alloc` silently converts a feature-flag check into a read of +uninitialized memory — this is a structurally hard bug to catch because it can pass on one GPU/driver +combination and fail (or silently diverge) on another. + +--- + +## 5. `vertvisc_type` pointer members: `Kv_shear`/`Kv_shear_Bu`/`MLD` as a recurring hazard class + +`vertvisc_type` (`MOM_variables.F90:258`) is the paradigm case of §1.3 (feature-gated pointers) and +has been the site of **three separate device-mapping bugs**, all variations on "the pointer's +`associated()` status and/or target contents were not correctly on device": + +1. **`d75e4870e` "vertvisc: Do not pass CS as pointers"** (Marshall Ward). Several + `MOM_vert_friction.F90` entry points (`vertvisc`, `vertvisc_remnant`, `vertvisc_limit_vel`) took + `type(vertvisc_CS), pointer :: CS` and began with an `if (.not.associated(CS)) call MOM_error(...)` + guard. The patch changes the dummy argument to a plain `type(vertvisc_CS) :: CS` (no `pointer`) + and drops the `associated()` guard (keeping only the `CS%initialized` logical check). Rationale + (from the commit message): *"This reduces some of the implicit 'microtransfers' required when + passing derived type point[er]s to and from the device."* Passing a CS **by pointer** into a + routine that then touches its members on device forces the compiler/runtime to re-resolve the + pointer's device address on every call (a "microtransfer" of the descriptor) instead of using a + plain by-reference/by-value derived-type argument whose device mapping was already established at + a higher scope. Net effect: same semantics, fewer descriptor round-trips per call. Also folded + into this commit: removed a redundant `!$omp target enter data map(alloc: b1, c1, d1, Ray, + b_denom_1)` / matching `exit data` pair around the tridiagonal solve in `vertvisc_remnant` — local + scratch arrays that don't need to persist across the call don't need explicit device + alloc/dealloc at all if they're firstprivate/private in the enclosing `target teams` region. + Guidance distilled: **once a CS's device mapping is established once (at `initialize_MOM`/child + `_init` time), pass it down the call tree as an ordinary (non-pointer) derived-type argument; only + the *owning* module should hold it as `pointer`/`allocatable` and manage its `enter + data`/`exit data` lifecycle.** + +2. **`c82e1254a` "vertvisc: Fix CS memory management"** (the most recent commit on `dev/gpu`, + in the log preceding this one). Root cause per commit message: *"The visc object was being + allocated twice, which was overwriting information about Kv_shear and Kv_shear_Bu that was + defined in `set_visc_register_restarts()`. This was causing errors in kernels which needed both + the arrays and the `associated()` state for flow control."* **Corrected mechanism (verified + against the diff and the parent tree):** the "allocated twice" is *not* a host double-`allocate`. + The host `allocate(CS%visc)` happens exactly once (`MOM.F90:3277`, unchanged by this commit). What + happened twice was the **device mapping** of the `visc` derived type: `!$omp target enter data + map(alloc: CS%visc)` at `MOM.F90:3278`, followed ~430 lines later by a redundant `!$omp target + enter data map(to: CS%visc, CS%set_visc_CSp)` at `:3709` (parent-commit line numbers). In between + those two, `set_visc_register_restarts` (called at `:3279`) `safe_alloc_ptr`'d and *device-attached* + `visc%Kv_shear`/`Kv_shear_Bu` (its own now-removed `!$omp target enter data map(alloc: + visc%Kv_shear)`). The second whole-struct `map(to: CS%visc)` at `:3709` **re-established / + overwrote the device image of the parent `visc` object**, clobbering those child pointer + attachments and the `associated()` state that had just been set up. Symptom: a device-side + "addressing error" in `double_gyre` runs, because subsequent `associated(visc%Kv_shear)` map guards + read a device descriptor that no longer matched the host object. The lesson is subtler and more + important than "don't allocate twice": **re-mapping a parent derived type with `map(to:)`/`map(alloc:)` + after its allocatable/pointer members have been individually device-attached can silently detach or + corrupt those member attachments — map the parent *once*, then attach members, and refresh the + parent's scalar contents with `target update to(...)`, never a second `enter data`.** Fix, in + three parts: + + > **Resolved (2026-07-14):** The root cause is re-allocated storage, not a refcount subtlety in + > nvfortran's present check. The host re-allocated `CS%visc`, so the second `map(to:)` targeted + > *different* storage than the first, orphaning the member attachments made in between. Two rules + > follow. First, "map the parent exactly once" guards against this regardless of which reading of + > the OpenMP spec you take. Second, a `map(to:)` on an already-present object does **not** refresh + > device contents — if a struct's host scalars or descriptors changed since its first map, the only + > refresh is `target update to(...)`. Never "re-map to refresh". + - `CS%set_visc_CSp` changed from embedded-by-value (`type(set_visc_CS) :: set_visc_CSp`) to + `type(set_visc_CS), allocatable :: set_visc_CSp` in `MOM.F90:417` (consistency with other + pointer/allocatable child CSs, and to make the allocate/deallocate lifecycle explicit and + single-sourced). + - `CS%visc` is updated to device *inside* `set_visc_init()` (`!$omp target update to(visc)`) right + after all its scalar members are finalized, **before** the conditional `Kv_shear`/`Kv_shear_Bu` + arrays are separately entered: + ```fortran + !$omp target update to(visc) + ... + !$omp target enter data map(to: visc%Kv_shear) if (associated(visc%Kv_shear)) + !$omp target enter data map(to: visc%Kv_shear_Bu) if (associated(visc%Kv_shear_Bu)) + ``` + - In `vertvisc_coef` (`MOM_vert_friction.F90`), `Kv_shear_Bu` handling changed from a **persistent + `map(alloc:)` at `set_viscosity_init` + a fresh `map(to:)`/`map(release:)` pair every call** to + the same `target update to(...) if (associated(...))` idiom already used for `Kv_shear` — i.e. + the fix explicitly *removed* an inconsistency where `Kv_shear` was persistently mapped and + refreshed with `target update`, but `Kv_shear_Bu` was instead re-entered/released every call + (`map(to: visc%Kv_shear_Bu) ... map(release: visc%Kv_shear_Bu)`), which is wasteful and, per the + surrounding comment removed in the diff, was based on a stale assumption (*"Kv_shear is + persistently mapped on device via map(alloc:) in set_viscosity_init, so map(to:) here would not + copy host updates"*) that no longer matched the actual mapping strategy once `Kv_shear_Bu` was + unified with the same pattern. + Lesson: **a pointer member that is both (a) conditionally allocated, (b) a restart-registry + target, and (c) persistently mapped to device is fragile against *any* re-mapping (or + re-allocation) of its owning derived type.** If the owning struct (`visc`) is re-mapped or + reallocated anywhere in the init sequence after its members are attached, every pointer inside it + must be re-established and re-mapped in the correct order: allocate host → `map(alloc:)` the parent + *once* → register restart (sets pointer) → `target update to` the parent's scalars/descriptors → + `map(to:)` the conditionally-present array members guarded by `associated()`. Never issue a second + whole-parent `enter data map(to:)`/`map(alloc:)` after the member attach step. + +3. `remotes/origin/vertvisc-no-ptr-transfer` (local remote branch, tip commit `f327f04c0`, subject + line identical to `d75e4870e`) is the same "do not pass CS as pointers" fix living on a + differently-based branch — evidence this fix was reapplied/rebased at least once, underscoring + that the CS-by-pointer anti-pattern recurs as code is merged from `dev-gfdl`. + +**Combined guidance for `vertvisc_type` and similarly-shaped types:** when a struct mixes always-on +`allocatable` arrays with feature-gated `pointer` arrays that are also restart targets, (1) allocate +the owning struct exactly once, ideally as `allocatable` at the top-level CS so its lifecycle is +unambiguous; (2) register restarts (which binds the pointers) before any device mapping of those +pointer members; (3) `target update to` the struct itself (its scalars + pointer descriptors) before +individually mapping/updating the conditionally-associated array members with an `if +(associated(...))` guard; (4) keep the guard identical in both the allocating routine (`set_visc_init`) +and every consuming routine (`vertvisc_coef`) — an asymmetric strategy (one map(alloc)-and-hold, the +other map(to)-and-release) is exactly what went wrong in point 2 above. + +--- + +## 6. Enumerated porting hazards (pointer-heavy structures, ranked) + +| Structure | File:line | Why hazardous | Status | +|---|---|---|---| +| `tracer_registry_type%Tr(MAX_FIELDS_)` | `MOM_tracer_types.F90:129`, `tracer_type` `:13` | AoS of ~20-pointer-member structs; `associated()` used as device control flow throughout `MOM_tracer_advect.F90`/`MOM_tracer_hor_diff.F90`; already caused the `map(alloc:)→map(to:)` bug (§4b) | Fixed on `dev/gpu` for advect path; watch any *new* code that maps `Reg%Tr(:)` | +| `ocean_internal_state` | `MOM_variables.F90:138` | All-pointer alias struct over the entire prognostic+accel state; populated once (`MOM.F90:3195-3212`) and handed to diagnostics/ensemble code — a large "view" object that must never itself be separately device-mapped (its members already are, via their true owners) | Not GPU-mapped itself (used for host-side diagnostics/ensembles); low risk if it stays that way | +| `vertvisc_type` (`MLD`, `Kd_shear`, `Kv_shear`, `Kv_shear_Bu`, `Kv_slow`, `TKE_turb`, `h_ML`, `sfc_buoy_flx`) | `MOM_variables.F90:258` | Feature-gated pointer + restart target + persistently mapped on device; three bugs already fixed here (§5) | Actively fixed/hardened; still fragile to any future re-map or re-allocation of `CS%visc` after its members are attached | +| `thermo_var_ptrs` (`tv%T`, `tv%S`, `tv%p_surf`) | `MOM_variables.F90:79` | Cross-module alias of the dycore's own `CS%T/CS%S`; re-established at two points (`MOM.F90:3119`, `:3432-3433`) — any code path that reallocates `CS%T`/`CS%S` without re-running the `tv%T => CS%T` assignment silently detaches `tv` from current data | No known bug yet, but structurally analogous to the `visc`/`Kv_shear` re-map bug (§5.2) — worth auditing anywhere `CS%T`/`CS%S` are reallocated (the alias must be re-run *and* the device image refreshed) | +| `accel_diag_ptrs` / `cont_diag_ptrs` (`ADp`, `CDp`) | `MOM_variables.F90:167,241` | All-pointer diagnostic aliases written by many producer modules (`vertvisc`, `CorAdCalc`, `PressureForce_FV`, `continuity`); each producer's `associated()` check on its own diagnostic slot must see a faithfully-mapped pointer | **`CS%ADp` *is* mapped `map(alloc: CS%ADp)` at `MOM.F90:3190`** (not `map(to:)`), and `associated(ADp%sal_u/tides_u/…)` is read as control flow in `MOM_PressureForce_FV.F90:913-931,2044-2058` — this is exactly the §4b shape. Safe today (host-side reads) but the lifecycle is incoherent — see the note below. | +| `MOM_restart_CS%var_ptrNd(:)` (`p0d..p4d`) | `MOM_restart.F90:130-134` | Heterogeneous pointer-array registry over arbitrary host arrays; host-only by design (I/O), but any future "GPU-resident restart" work would hit the same AoS-of-pointers attach cost documented in `1865612de` | Host-only today (`00-architecture.md` §7.4); a future hazard, not a current one | +| Any *future* `type(p2d)/type(p2di) dimension(SZJ_(G))` (array-of-pointer-to-2D-array, one alloc per row) | pattern retired in `MOM_tracer_hor_diff.F90` by `1865612de`, defined at `:104-110` (now dead code — no remaining users in that file, verified) | Per-row `enter data` inside a loop is the concrete anti-pattern that cost 2x compile/attach time; the type definitions remain in-file as a fossil/warning | Fixed here; **do not reintroduce this pattern elsewhere** (e.g. `MOM_set_diffusivity.F90`, `MOM_CVMix_KPP.F90`, `MOM_energetic_PBL.F90` are still unported and may contain the same idiom — check before porting) | + +> **Resolved (2026-07-14):** Not a latent `Reg%Tr(:)`-style bug — the `associated(ADp%…)` reads in +> `PressureForce_FV` sit in plain host loops, where the host descriptor is authoritative. But the +> `ADp` mapping lifecycle is internally inconsistent. `initialize_MOM` maps `CS%ADp` with +> `map(alloc:)` (refcount 1, a garbage shell); the first `vertvisc` (`MOM_vert_friction.F90`) does +> `enter data map(to: ADp)` on an already-present object, so the refcount goes to 2 and **the `to` +> copy is skipped** — the shell stays garbage, and only the explicitly attach-mapped +> `du_dt_str`/`dv_dt_str` get valid device descriptors, which is the sole reason `vertvisc`'s +> device-side `associated(ADp%…)` reads are safe. The matching `exit data map(delete: ADp)` then +> forces the refcount to 0, destroying `initialize_MOM`'s mapping; every later `vertvisc` call +> re-creates the shell fresh, now with a real `to` copy. Net: the init-time map is dead weight. +> Fix (maintainer's choice): either drop the init-time map and let `vertvisc` own the per-call +> lifecycle with `release`, or make the init-time map authoritative (`map(to:)` + per-call +> `update to(ADp)`, no per-call delete). Do **not** `map(to:)` the shell in `initialize_MOM` — that +> was proposed and is wrong. + +--- + +## 7. Summary of concrete guidance for a porting agent + +1. Before mapping any derived-type instance or array-of-derived-types to device, ask: **does any + code path read `associated()` on one of its pointer members, or a scalar member, from inside + device code?** If yes, it must be `map(to:)` (or `target update to`), never `map(alloc:)` + (§4b, exemplified by `a774eb331`). +2. **Never write to a shared array element from inside a `do concurrent` without a `DO_LOCALITY` + reduction clause** — use a scalar temporary and assign the array element once afterward (§4a, + `e182de310`). +3. **Don't pass a CS as `pointer` into leaf routines just to check `associated(CS)`** — pass by + ordinary derived-type argument once its device mapping is established at a higher scope; this + avoids repeated pointer-descriptor "microtransfers" (§5.1, `d75e4870e`). +4. **Map a parent derived type to device exactly once, then attach its members; never re-map the + parent afterward.** A second whole-parent `enter data map(to:)`/`map(alloc:)` issued after its + allocatable/pointer members are individually device-attached clobbers those attachments (§5.2, + `c82e1254a` — the "allocated twice" was a *double device mapping* at `MOM.F90:3278`+`:3709`, not a + host double-`allocate`). Correct order: allocate host → `map(alloc:)` parent once → register + restart (binds pointers) → `target update to` the parent's scalars/descriptors → `map(to:)` the + pointer members guarded by `associated()`. +5. **Match the guard intrinsic to the member kind:** `if (allocated(...))` for allocatable members, + `if (associated(...))` for pointer members (`vertvisc_type` uses both — `MOM.F90:1807-1812` vs the + `Kv_shear` guards). Mixing them is a compile error or silent control-flow bug (§3). +6. **Flatten arrays-of-pointer-structs on hot paths** (`type(p2d/p2di), dimension(SZJ_(G))`) to flat + allocatable arrays — the per-row attach/detach cost is real and measured (`1865612de`, ~2x compile + time saved for ~20% more memory). +7. Pointers whose only job is a short local alias to a macro-allocatable CS member (`u => CS%u`, + `eta => CS%eta`) are not aliasing hazards in the mapping sense — but they're not free either, since + a compiler doing alias analysis for `do concurrent`/`target` regions has to prove non-aliasing + through them. Retire them opportunistically when a routine is otherwise being touched, not as a + dedicated pass. **But** distinguish these from pointers whose target is *reassigned* at runtime + (`p_surf`, `u_ptr`/`v_ptr` feeding `btstep`): those are load-bearing and must keep their + per-call `associated()`-guarded map (§1.4, §3). + +--- + +## 8. Verification notes + +Every commit reconstruction and every file:line citation in this doc was checked against the actual +source and `git show`/`git diff` on branch `dev/gpu`. + +**Confirmed correct as written:** +- `e182de310` (§4a, `domore_k` write-write race → scalar-temp + `reduce(max:)`): diff matches exactly, + including the removal of `domore_k` from the map lists and the three call sites. +- `a774eb331` (§4b, `Reg%Tr(:)` `map(alloc:)`→`map(to:)`): diff and the paired `exit data` + (`map(from: hprev)`→`map(release: hprev, …)`) match; `map(to: Reg%Tr(m)%t)` at `:237` and the + `associated(Reg%Tr(m)%…)` guards at `:244-266` confirmed. +- `d75e4870e` (§5.1, "do not pass CS as pointers"): all three signatures (`vertvisc`, + `vertvisc_remnant`, `vertvisc_limit_vel`) changed from `pointer` to plain `type(vertvisc_CS)`, the + `associated(CS)` guards dropped, and the redundant `map(alloc: b1,c1,d1,Ray,b_denom_1)`/`map(delete:)` + pair around the `vertvisc_remnant` tridiagonal solve removed — all confirmed. +- `1865612de` (§2, AoS-of-pointers flattening): commit message figures (~700k→830k, ~20%, "halves" + compile time) quoted verbatim; pre-commit form `type(p2d), dimension(SZJ_(G)) :: deep_wt_Lu` with + per-`j` `allocate(...(j)%p(...))` + in-loop `map(alloc: deep_wt_Lu(J)%p, …)` confirmed at parent; the + `p2d`/`p2di` types are now genuinely dead (no users) — confirmed. +- All `MOM_variables.F90` type line numbers (`:79/:138/:167/:241/:258/:317`), the `tv%T => CS%T` + assignments (`MOM.F90:3119`, `:3432-3433`), the `ocean_internal_state` population block + (`MOM.F90:3195-3212`), `register_restart_field_ptr3d` (`MOM_restart.F90:207-239`), `var_ptrNd` + (`:130-134`), `tracer_type`/`tracer_registry_type` (`:13/:15/:129`), `MAX_FIELDS_=50`, and the + `p_surf` local-pointer resolution (`MOM_dynamics_split_RK2.F90:461-467`, mapped `:472`, released + `:939`) — all confirmed. + +**Corrected:** +1. **§1.4 (incidental pointers):** the draft's dycore example `u,v,h => NULL()` "declared at `:151-153`, + bound at `:165`" was fabricated — `:151-153` are CS-type members (`taux_bot`/`tauy_bot`) and no + `u => CS%u` binding exists in `MOM_dynamics_split_RK2.F90`. Replaced with the real aliases + (`u_av/v_av/h_av => CS%… ; eta => CS%eta` at `:426`) and noted `u,v,h` arrive as dummy arguments. + Added the contrasting *reassigned-alias* class (`u_ptr`/`v_ptr`/`p_surf`). +2. **§5.2 (`c82e1254a`):** the draft called this "a plain host-side double-`allocate` bug." That is + wrong — the host `allocate(CS%visc)` occurs once (`MOM.F90:3277`, untouched by the commit). The real + defect is a **double *device mapping*** of the parent struct: `map(alloc: CS%visc)` at `:3278` then a + redundant `map(to: CS%visc)` at `:3709` (parent line numbers), the second of which overwrote the + device image and clobbered the `Kv_shear`/`Kv_shear_Bu` attachments created in between by + `set_visc_register_restarts`. Rewrote the mechanism and the derived rule accordingly (map parent + once, never re-`enter data`). + +**Enhancements added:** the `allocated()` vs `associated()` guard-intrinsic distinction (§3, with +`MOM.F90:1807-1812`); the reassigned-alias hazard class (§1.4); tightened §7 rules (now 7 rules); and +a flagged latent-hazard finding that `CS%ADp` is itself mapped `map(alloc:)` at `MOM.F90:3190`. + +**Confidence:** High on all commit reconstructions and line numbers (directly verified against +source/git). The §5.2 clobber mechanism and the `CS%ADp` mapping question are both settled — see the +resolved notes in those sections. diff --git a/knowledge/gpu-knowledge/03-openmp-mapping.md b/knowledge/gpu-knowledge/03-openmp-mapping.md new file mode 100644 index 0000000..fa2655c --- /dev/null +++ b/knowledge/gpu-knowledge/03-openmp-mapping.md @@ -0,0 +1,689 @@ +# OpenMP Target Data-Mapping Infrastructure (dev/gpu) + +> **Purpose.** Inventory of the *proven-works* patterns for making MOM6 control-structure (CS) +> members and local scratch arrays device-resident under OpenMP target offload. This is a "what +> works" catalogue for extending the port to new modules — not a tutorial on OpenMP itself. Read +> `00-architecture.md` §2, §6, §7.3 first. + +**Directive counts across `src/` + `config_src/`** (reproduced by +`grep -rn "omp target enter data\|omp target exit data\|omp target update\|omp declare target\|map(" src/ config_src/`): + +| Directive | Count | +|---|---| +| `!$omp target enter data` | 213 | +| `!$omp target exit data` | 168 | +| `!$omp target update` | 398 | +| `!$omp declare target` | 21 | + +--- + +## 1. The canonical CS-member lifecycle + +The dominant, hand-written idiom co-locates the mapping directive directly next to the +`ALLOC_`/`DEALLOC_` macro call (`MOM_memory_macros.h`) that creates/destroys the host array. There +is **no wrapper macro or subroutine** for this — see §4. + +### 1.1 Allocate → enter data (init routine) + +`src/core/MOM_dynamics_split_RK2.F90:1350-1368` (`register_restarts_dyn_split_RK2`): + +```fortran +! TODO: Are these initializations necessary? If not, then we can do +! map(alloc:) rather than map(to:) +ALLOC_(CS%diffu(IsdB:IedB,jsd:jed,nz)) ; CS%diffu(:,:,:) = 0.0 +ALLOC_(CS%diffv(isd:ied,JsdB:JedB,nz)) ; CS%diffv(:,:,:) = 0.0 +!$omp target enter data map(to: CS%diffu, CS%diffv) +ALLOC_(CS%CAu(IsdB:IedB,jsd:jed,nz)) ; CS%CAu(:,:,:) = 0.0 +ALLOC_(CS%CAv(isd:ied,JsdB:JedB,nz)) ; CS%CAv(:,:,:) = 0.0 +!$omp target enter data map(to: CS%CAu, CS%CAv) +... +ALLOC_(CS%eta(isd:ied,jsd:jed)) ; CS%eta(:,:) = 0.0 +ALLOC_(CS%u_av(IsdB:IedB,jsd:jed,nz)) ; CS%u_av(:,:,:) = 0.0 +ALLOC_(CS%v_av(isd:ied,JsdB:JedB,nz)) ; CS%v_av(:,:,:) = 0.0 +ALLOC_(CS%h_av(isd:ied,jsd:jed,nz)) ; CS%h_av(:,:,:) = GV%Angstrom_H +!$omp target enter data map(to: CS%eta, CS%u_av, CS%v_av, CS%h_av) +``` + +Note the host-side zero-init (`CS%diffu(:,:,:) = 0.0`) *before* the map — `map(to:)` copies that +initialized value up. The in-source `TODO` at line 1350 flags that this is sometimes wasted work +where `map(alloc:)` (no copy) would do — see §3 kind-selection rules. + +### 1.2 Matching exit data → deallocate (`*_end` routine) + +`src/core/MOM_dynamics_split_RK2.F90:2065-2083` (`end_dyn_split_RK2`) — directives appear in +**reverse order** of the enter-data calls, immediately paired with `DEALLOC_`: + +```fortran +DEALLOC_(CS%diffu) ; DEALLOC_(CS%diffv) +!$omp target exit data map(delete: CS%diffu, CS%diffv) +DEALLOC_(CS%CAu) ; DEALLOC_(CS%CAv) +!$omp target exit data map(delete: CS%CAu, CS%CAv) +DEALLOC_(CS%CAu_pred) ; DEALLOC_(CS%CAv_pred) +!$omp target exit data map(delete: CS%CAu_pred, CS%CAv_pred) +DEALLOC_(CS%PFu) ; DEALLOC_(CS%PFv) +!$omp target exit data map(delete: CS%PFu, CS%PFv) +... +DEALLOC_(CS%eta) ; DEALLOC_(CS%eta_PF) ; DEALLOC_(CS%pbce) +!$omp target exit data map(delete: CS%eta, CS%eta_PF, CS%pbce) +DEALLOC_(CS%h_av) ; DEALLOC_(CS%u_av) ; DEALLOC_(CS%v_av) +!$omp target exit data map(delete: CS%u_av, CS%v_av, CS%h_av) +``` + +This enter/exit pair (register-restarts ↔ end) is the module-lifetime idiom. A second, shorter-lived +idiom wraps a **single subroutine call**: allocate/enter-data at entry, exit-data/deallocate at +return, for pure scratch (non-CS) arrays — see §1.3. + +### 1.3 Local-scratch (non-CS, subroutine-scoped) lifecycle + +Two flavours: + +- **Stack-declared automatic arrays**, entered/exited around their live range inside one routine — + `src/core/MOM_continuity_PPM.F90:181` / `:228` (`continuity_PPM`): + + ```fortran + !$omp target enter data map(alloc: h_W, h_E, h_S, h_N) + ... ! zonal/meridional edge reconstruction + mass flux calls + !$omp target exit data map(delete: h_W, h_E, h_S, h_N) + end subroutine continuity_PPM + ``` + + and `MOM_continuity_PPM.F90:658-660` (`zonal_mass_flux`, multi-line continuation): + + ```fortran + !$omp target enter data & + !$omp map(alloc:uhbt_t,uh_t,duhdu,du,du_min_CFL,du_max_CFL,duhdu_tot_0,uh_tot_0, & + !$omp visc_rem_max,do_I,visc_rem,simple_OBC_pt) + ``` + +- **Dummy-argument scratch arrays owned by the caller but mapped by the callee's caller** — + `MOM_dynamics_split_RK2.F90:435-436` (`step_MOM_dyn_split_RK2`, entered right after locals are + declared, before any use): + + ```fortran + !$omp target enter data map(alloc: u_bc_accel, v_bc_accel, eta_pred, uh_in, vh_in) + !$omp target enter data map(alloc: up, vp, hp, dz, h_tmp) + ``` + + balanced at the very end of the same subroutine, `:1193-1194`, using **two different map kinds** + for the two groups: + + ```fortran + !$omp target exit data map(release: u_bc_accel, v_bc_accel, eta_pred, uh_in, vh_in) + !$omp target exit data map(delete: hp, up, vp, dz, h_tmp) + ``` + + (`up, vp` are additionally deleted early, mid-routine, at `:1253`, once their last use in the + corrector has passed — an example of shrinking device residency lifetime below the whole + subroutine when memory pressure or reuse patterns warrant it.) + +### 1.4 Balance bugs and their fixes (git history) + +The enter/exit pairing is maintained **by hand** and has drifted out of sync more than once: + +- **`15ca2a25f` "vertvisc: add missing b_denom_1 map delete"** — `src/parameterizations/vertical/MOM_vert_friction.F90`. + The enter-data list `map(alloc: b1, c1, d1, Ray, b_denom_1)` was *replaced* (not matched) by a + narrower `map(delete: b1, c1, d1, Ray)` at teardown, silently leaking `b_denom_1`'s device + allocation every call. Fix: mirror the full original list: + + ```diff + - !$omp target enter data map(alloc: b1, c1, d1, Ray, b_denom_1) + !$omp target enter data map(to: visc%Ray_v) if (allocated(visc%Ray_v)) + ... + - !$omp target exit data map(delete: b1, c1, d1, Ray) + + !$omp target exit data map(delete: b1, c1, d1, Ray, b_denom_1) + ``` + + (The enter-data line for `b1,c1,d1,Ray,b_denom_1` itself was later removed as redundant — it + duplicated an allocation done elsewhere in the routine — but the lesson generalizes: any edit that + changes an enter-data variable list must grep the same routine for the paired exit-data list.) + +- **`bc05a6a89` "Explicitly allocate h_tmp to prevent expensive transfers during initialization"** — + `MOM_dynamics_split_RK2.F90`. Before this commit `h_tmp` (an automatic/scratch array) had **no** + enter/exit data pair at all in `initialize_dyn_split_RK2`, so every `do concurrent` touching it + paid an implicit per-loop host-device transfer (or relied on unified memory). Fix bracketed its + one use site with an explicit pair and converted the surrounding manual triple-nested loops to + `do concurrent`: + + ```fortran + !$omp target enter data map(alloc: h_tmp ) + if (CS%store_CAu) then + ... + do concurrent (k=1:nz, j=jsd:jed, i=isd:ied) + h_tmp(i,j,k) = h(i,j,k) + enddo + call continuity(CS%u_av, CS%v_av, h, h_tmp, uh, vh, dt, G, GV, US, CS%continuity_CSp, CS%OBC, pbv) + ... + endif + !$omp target exit data map(delete: h_tmp ) + ``` + + Generalizable rule: **any array touched inside a `do concurrent`/`omp target` region must have an + explicit enter-data before the first touch**, even if it is only "temporary" scratch — omitting it + does not fail to compile, it silently reintroduces per-statement host-device traffic. + +**Practical balance-checking method used in this codebase:** grep the variable name for both +`enter data` and `exit data` inside the same subroutine (see §1.3's `up, vp` example: entered +*once* at `:436` — `map(alloc: up, vp, hp, dz, h_tmp)` — but deleted in *two* separate exit-data +statements, at `:1194` — `map(delete: hp, up, vp, dz, h_tmp)` — and again at `:1253` — +`map(delete: up, vp)`. The extra early delete at `:1253` is intentional early release once the +corrector's last use of `up, vp` has passed, not a bug — but this is exactly the kind of +one-enter-vs-two-exit count mismatch a balance check must be able to *explain* rather than flag +blindly). Because a second `map(delete:)` on an already-removed variable is a no-op under +nvfortran, this double-delete is safe; the danger a balance check guards against is the opposite — +an enter with no matching exit (the `15ca2a25f` leak). + +--- + +## 2. Mapping whole derived-type shells before members ("partial presence") + +nvfortran (like most OpenMP implementations) requires a struct's own memory to be "present" on +device before any of its allocatable/pointer members can be attached. The codebase's fix is to map +the **bare CS shell** with `map(alloc:)` *before* calling the child module's `_init` routine (which +then maps its own members internally): + +> **Mechanism (attach/detach under nvfortran).** When you `map(to:/alloc:)` a derived-type variable, +> only its *scalar* fields and its descriptor words travel — an allocatable/pointer array **member** +> is a separate device allocation whose device descriptor must then be *attached* (pointed) to the +> parent's device copy. That attach only happens if the parent shell is already present, which is why +> the shell map must strictly precede the member maps (violating the order is the "ambiguous partial +> presence" error `81680c15d` fixed). Corollary rules an agent can apply mechanically: (1) map order +> is always **outermost shell → … → innermost array**, and teardown is the exact reverse; (2) each +> attach is a per-member device operation, so mapping an *array of* member-bearing structs one element +> at a time is O(#elements) attaches — the cost `1865612de` eliminated by flattening (§3.4); (3) a +> plain scalar CS member (an `id_*` diagnostic ID, a block size) needs **no** separate map — it rides +> along inside the shell's `map`, and is refreshed with whole-struct `update to(CS)` (§2.2), never its +> own `enter data`. + +`src/core/MOM_dynamics_split_RK2.F90:1689-1712` (`initialize_dyn_split_RK2`): + +```fortran +!$omp target enter data map(alloc: CS%continuity_CSp) +call continuity_init(Time, G, GV, US, param_file, diag, CS%continuity_CSp, CS%OBC) +... +!$omp target enter data map(alloc: CS%PressureForce_CSp) +call PressureForce_init(Time, G, GV, US, param_file, diag, CS%PressureForce_CSp, CS%ADp, & + CS%SAL_CSp, CS%tides_CSp) + +!$omp target enter data map(alloc: CS%hor_visc) +call hor_visc_init(Time, G, GV, US, param_file, diag, CS%hor_visc, ADp=CS%ADp) + +allocate(CS%vertvisc_CSp) +!$omp target enter data map(alloc: CS%vertvisc_CSp) +call vertvisc_init(MIS, Time, G, GV, US, param_file, diag, CS%ADp, dirs, & + ntrunc, CS%vertvisc_CSp, CS%fpmix) +... +!$omp target enter data map (alloc: CS%barotropic_CSp) +call barotropic_init(u, v, h, Time, G, GV, US, param_file, diag, & + CS%barotropic_CSp, restart_CS, calc_dtbt, CS%BT_cont, & + CS%OBC, CS%SAL_CSp, HA_CSp) +``` + +The pointer-typed child (`vertvisc_CSp`) needs an explicit host `allocate(CS%vertvisc_CSp)` first +(pointers have no storage until allocated), whereas the by-value children (`hor_visc`, +`continuity_CSp`, `PressureForce_CSp`, `barotropic_CSp` — embedded `type(x_CS) :: x` members) already +have storage as part of the parent and only need the shell attached. + +**Top-level analogue** in `MOM.F90` — the whole `MOM_control_struct` and both viscosity CSs, added by +commit `81680c15d` ("Allocate MOM CS and both viscosity CS on GPU") specifically to fix "ambiguous +partial presence" errors: + +```fortran +! config_src/drivers/solo_driver/MOM_driver.F90 +!$omp target enter data map(alloc: MOM_CSp) +... +call MOM_end(MOM_CSp) +!$omp target exit data map(delete: MOM_CSp) +``` + +```fortran +! src/core/MOM.F90, initialize_MOM — allocate(CS%visc) is now required because vertvisc_type +! was changed from an embedded value member to `type(vertvisc_type), allocatable :: visc` +allocate(CS%visc) +!$omp target enter data map(alloc: CS%visc) +call set_visc_register_restarts(HI, G, GV, US, param_file, CS%visc, restart_CSp, use_ice_shelf) +``` + +Commit `81680c15d` also changed `G_in` (`ocean_grid_type`) from an embedded value member to +`allocatable`, "to prevent excessive grid transfers" — i.e. the shell-map pattern only works cleanly +when the member is a pointer/allocatable; embedded-by-value derived-type members inside another +mapped struct are harder to manage independently and were converted to allocatable specifically to +decouple their device lifetime from the parent's. + +### 2.1 Nested sub-types: `CS%pbv` and its members + +`src/core/MOM.F90:3225-3231` — a two-level nest (`MOM_control_struct` → `porous_barrier_type pbv` → +four allocatable arrays), mapped shell-then-members in one block, non-conditionally: + +```fortran +allocate(CS%pbv%por_face_areaU(IsdB:IedB,jsd:jed,nz), source=1.0) +allocate(CS%pbv%por_face_areaV(isd:ied,JsdB:JedB,nz), source=1.0) +allocate(CS%pbv%por_layer_widthU(IsdB:IedB,jsd:jed,nz+1), source=1.0) +allocate(CS%pbv%por_layer_widthV(isd:ied,JsdB:JedB,nz+1), source=1.0) +!$omp target enter data map(to: CS%pbv) +!$omp target enter data map(to: CS%pbv%por_face_areaU, CS%pbv%por_face_areaV) +!$omp target enter data map(to: CS%pbv%por_layer_widthU, CS%pbv%por_layer_widthV) +``` + +Order matters: `CS%pbv` (the shell) must be entered before `CS%pbv%por_face_areaU` etc. (the +members), exactly mirroring the parent-CS-then-child-CS ordering in §2's dycore example — this is +the same rule applied one level deeper. + +### 2.2 Whole-struct bulk `update` + +Rather than updating individual scalar members one at a time after a batch of host-side parameter +computation, the codebase sometimes updates the **entire mapped CS** in one directive: + +`src/core/MOM_barotropic.F90:6576` (`barotropic_init`, after ~30 `register_diag_field` calls that +set scalar `CS%id_*` diagnostic-ID members) and `:6588`: + +```fortran +!$omp target update to (CS) +... +!$omp target enter data map (to: CS%frhatu, CS%frhatv) +!$omp target enter data map (to: CS%eta_cor) +call set_dtbt(G, GV, US, CS, gtot_est=gtot_estimate, SSH_add=SSH_extra) +... +!$omp target update to (CS%dtbt) +``` + +and `src/core/MOM.F90:3104` (`initialize_MOM`, right after `CS%G_in`'s grid metrics have been +uploaded, and again after `set_visc_init` in `MOM_set_viscosity.F90` at the analogous `CS` update +site introduced by `81680c15d`): + +```fortran +call tracer_registry_init(param_file, CS%tracer_Reg) + +!$omp target update to(CS) +``` + +The commit message for `81680c15d` flags this as an experimental departure from the codebase's usual +member-by-member discipline: *"one change here breaks our derived type handling rules: we do an +`update(CS)` after the grid has been uploaded... this needs exploration."* Treat whole-struct +`update to(CS)` as a **pragmatic escape hatch** used when a batch of scalar CS members (mostly +diagnostic IDs / dtbt-like scalars) must reach the device and enumerating each one is impractical, +not as the general convention — the general convention is per-array `map`/`update` next to each +`ALLOC_`/`DEALLOC_`. + +--- + +## 3. Proven-works mapping pattern catalogue + +### 3.1 Macro-allocatable array inside a CS (the dominant pattern) + +See §1.1/§1.2 in full. One-line summary: `ALLOC_(CS%x(...)) ; CS%x = ` then +`!$omp target enter data map(to: CS%x)` in `*_init`; `DEALLOC_(CS%x)` then +`!$omp target exit data map(delete: CS%x)` in `*_end`, same order both directions is not required but +strongly conventional. + +### 3.2 Nested CS shell ("partial presence") + +See §2 in full — `map(alloc: CS%child_CSp)` before calling `child_init`, both for pointer children +(needs `allocate()` first) and by-value embedded children. + +### 3.3 Conditional maps: `if(associated())` / `if()` + +Two distinct places the `if` can go — as a Fortran `if` block around the whole directive, or as an +OpenMP `if()` clause on the directive itself (compiled unconditionally, skipped at runtime): + +- OpenMP `if()` clause, `src/core/MOM_dynamics_split_RK2.F90:472` / `:939` (`p_surf` is a pointer + that may alias either `p_surf_end` or `forces%p_surf`, and is only sometimes associated): + + ```fortran + !$omp target enter data map(to: p_surf) if (associated(p_surf)) + ... + !$omp target exit data map(delete: p_surf) if (associated(p_surf)) + ``` + +- Same idiom for a hybrid allocatable/pointer CS member. The **live** current example is the pair + of guarded whole-field updates at `src/parameterizations/vertical/MOM_vert_friction.F90:1440-1441`: + + ```fortran + !$omp target update to(visc%Kv_shear) if (associated(visc%Kv_shear)) + !$omp target update to(visc%Kv_shear_Bu) if (associated(visc%Kv_shear_Bu)) + ``` + + (Historically the same routine also carried + `!$omp target enter data map(to: visc%Ray_v) if (allocated(visc%Ray_v))` — visible in the diff + context of commit `15ca2a25f`. That directive has since been **removed**: in current source + `visc%Ray_v` is read pointwise inside the tridiagonal loop with an inline + `if (allocated(visc%Ray_v)) Ray = visc%Ray_v(i,J,k)` guard — `MOM_vert_friction.F90:942,954,1259,1267` + — and the associated `b1,c1,d1,Ray,b_denom_1` scratch became loop-`private(...)` rather than + device-mapped. Do not cite `:437` for a `map` — line 437 is the `!$omp declare target` of + `find_coupling_coef_gl90`.) + + `vertvisc_type` (`MOM_variables.F90:258`) is documented in `00-architecture.md` §2.3 as **hybrid** + — some fields allocatable, some pointer (because they must be restart-registry targets) — so every + map of one of its optional fields must be guarded (`allocated()` for allocatable fields, + `associated()` for pointer fields), since the field may legitimately be unallocated for a given + configuration (e.g. `Kv_shear` only exists if KPP/shear mixing is active). + +- Fortran `if` block guarding a diagnostic-ID-gated device→host round trip (see §6), + `src/tracer/MOM_tracer_hor_diff.F90:665-680`: + + ```fortran + if (CS%id_KhTr_u > 0) then + !$omp target exit data map(from: Kh_u) + do j=js,je ; do I=is-1,ie + Kh_u(I,j,:) = G%mask2dCu(I,j)*Kh_u(I,j,1) + enddo ; enddo + ... + call post_data(CS%id_KhTr_u, Kh_u, CS%diag) + endif + ``` + +### 3.4 Flat array vs. array-of-structs ("struct of arrays" pitfall) + +**Anti-pattern (pre-fix):** an array of small derived types each holding its own pointer/allocatable +member (`type(p2d), dimension(SZJ_(G)) :: deep_wt_Lu` — `p2d` wraps a single `real, pointer :: p(:,:)` +component), mapped **one struct at a time inside a `do j` loop** — commit `cdd3de9e3` ("add data +mapping for tracer_epipycnal_ML_diff"), `src/tracer/MOM_tracer_hor_diff.F90` (pre-flatten form): + +```fortran +do j=js,je + k_size = max(2*max_srt(j),1) + allocate(deep_wt_Lu(j)%p(IsdB:IedB,k_size)) + ... + !$omp target enter data map(alloc: deep_wt_Lu(J)%p, deep_wt_Ru(J)%p, hP_Lu(J)%p, hP_Ru(J)%p, & + !$omp k0a_Lu(j)%p, k0a_Ru(j)%p, k0b_Lu(j)%p, k0b_Ru(j)%p) +enddo +``` + +This requires one `attach`/`detach` operation *per j-row, per array* — `SZJ_(G)` separate small +device allocations and pointer attachments instead of one big one. + +**Fix (proven pattern):** commit `1865612de` ("Convert structs of arrays to flat arrays") — replace +`type(p2d), dimension(SZJ_(G)) :: deep_wt_Lu` with a single +`real, dimension(:,:,:), allocatable :: deep_wt_Lu` (index order `(I,k,j)`), sized once by a +`do concurrent ... DO_LOCALITY(reduce(max:k_size))` over all rows, allocated and mapped **once**: + +```fortran +k_size = 1 +do concurrent (j=js-1:je+1) DO_LOCALITY(reduce(max:k_size)) + k_size = max(k_size, 2*max_srt(j)) +enddo +allocate(k0a_Lu(IsdB:iedB,k_size,jsd:jed)) +allocate(k0a_Ru(IsdB:iedB,k_size,jsd:jed)) +allocate(deep_wt_Lu(IsdB:iedB,k_size,jsd:jed)) +allocate(deep_wt_Ru(IsdB:iedB,k_size,jsd:jed)) +... +!$omp target enter data map(alloc: deep_wt_Lu, deep_wt_Ru, hP_Lu, hP_Ru, k0a_Lu, k0a_Ru, k0b_Lu, & +!$omp k0b_Ru) +``` + +Measured effect (commit message, `1865612de`): **halved GPU compile-region time in +`MOM_tracer_hor_diff`**, at the cost of ~20% more memory (worst-case per-array element count rose +from ~700k to ~830k, because every `j`-row now allocates the same `k_size` instead of its own +tighter `max(2*max_srt(j),1)`). **Rule of thumb for new ports: never map a derived-type array whose +element is itself a pointer/allocatable-holding struct inside a loop — flatten to one contiguous +array with the loop index as a trailing dimension first.** + +**Accepted, unfixed instance of the same anti-pattern:** the tracer registry `Reg%Tr(:)` +(`tracer_type`, one array element per tracer, each independently shaped/sized) is *not* flattened — +each element's members are mapped individually inside a host loop over tracers, using the `!$` +free-form conditional-compilation sentinel (compiled only when OpenMP is enabled, so the loop +variable `m` doesn't need to exist in a non-OpenMP build) — commit `97629c240`, +`src/tracer/MOM_tracer_hor_diff.F90:207-210`: + +```fortran +! MOM_tracer_hor_diff.F90:209-216 +!$omp target enter data map(to: Reg, Reg%Tr, CS) map(alloc: khdt_x, khdt_y, kh_u, kh_v) +!$ do m = 1, Reg%ntr + !$omp target enter data map(to: Reg%Tr(m)%t) + !$omp target enter data map(to: Reg%Tr(m)%df_x) if(associated(Reg%Tr(m)%df_x)) + !$omp target enter data map(to: Reg%Tr(m)%df_y) if(associated(Reg%Tr(m)%df_y)) + !$omp target enter data map(to: Reg%Tr(m)%df2d_x) if(associated(Reg%Tr(m)%df2d_x)) + !$omp target enter data map(to: Reg%Tr(m)%df2d_y) if(associated(Reg%Tr(m)%df2d_y)) +!$ enddo +``` + +Note how the mandatory `Reg%Tr(m)%t` field is mapped unconditionally while every *optional* pointer +field (`df_x`, `df_y`, `df2d_x`, `df2d_y`) gets its own `if(associated(...))`-guarded directive — the +per-element/per-field guarding is precisely what makes this awkward to flatten. Also note the +`map(to: Reg, Reg%Tr, CS)` on the first line: the registry shell **and** the `Reg%Tr(:)` array of +`tracer_type` must be present before any `Reg%Tr(m)%...` member can attach, the same shell-before-member +ordering as §2. This is tolerated (rather than flattened like `deep_wt_Lu` in +`tracer_epipycnal_ML_diff`) because `ntr` and each tracer's presence of optional fields vary +per-configuration and per-element — flattening would require a redesign of +`tracer_type` itself, a larger change than the local scratch-array flattening in `1865612de`. It +remains a candidate for the same fix if the registry loop is ever shown to dominate profile time. + +### 3.5 `map(alloc)` vs `map(to)` vs `map(from)` vs `map(delete)` vs `map(release)` — when each is used + +| Kind | Used when | Example | +|---|---|---| +| `map(to:)` | Host has meaningful initial data the device kernel reads before ever writing it (zeroed/`GV%Angstrom_H`-initialized CS arrays, grid metrics, restart-read fields) | `MOM_dynamics_split_RK2.F90:1354` `map(to: CS%diffu, CS%diffv)` (after explicit host zero-init) | +| `map(alloc:)` | Device-only scratch, or a CS array whose first write happens on-device and host content is irrelevant (the `MOM_dynamics_split_RK2.F90:1350` `TODO` explicitly asks "if not [needed], do `map(alloc:)` rather than `map(to:)`") | `MOM_dynamics_split_RK2.F90:1637` `map(alloc: CS%uhbt, CS%vhbt)`; nested CS shells (§2) always use `alloc` | +| `map(from:)` | One-shot device→host copy-out, typically for a diagnostic about to be posted or a value about to feed host-only code | `MOM_tracer_hor_diff.F90:666` `map(from: Kh_u)` right before `post_data` | +| `map(delete:)` | Paired teardown of a `map(to:)/map(alloc:)` at `*_end`/end-of-scope. Forces the device reference count to **zero** and deallocates regardless of prior count — **does not copy back** | `MOM_dynamics_split_RK2.F90:2066` `map(delete: CS%diffu, CS%diffv)` | +| `map(release:)` | Teardown that **decrements** the device reference count by one (deallocating only if it hits zero) — also does not copy back. Used where the mapped variable may have been mapped from more than one place, or where the exact map state across conditional branches is harder to track statically | `MOM_dynamics_split_RK2.F90:1193` `map(release: u_bc_accel, v_bc_accel, eta_pred, uh_in, vh_in)`; `MOM_tracer_hor_diff.F90:723` `map(release: khdt_x, khdt_y, Kh_u, Kh_v) map(release: CS)` | + +**Critical copyback rule:** neither `delete` nor `release` copies the device value back to the host — +they only tear down the device allocation. Any array whose *final host value matters* after the device +region (e.g. it will be written to a restart, read by host-only code, or checksummed) must be brought +back with `!$omp target update from(...)` or `!$omp target exit data map(from:)` **before** the +`delete`/`release`. In this codebase, CS work arrays like `CS%diffu` are pure device-side intermediates +recomputed every timestep, so `map(delete:)` with no copy-back is correct; the diagnostic/restart +copy-outs are handled separately by the `map(from:)`/`update from` sites in §6. + +Practical distinction actually driving the choice in this codebase between `delete` and `release`: +`delete` is used for **CS members**, where the enter/exit pairing is unconditional and exactly +mirrored (§1.2); `release` is used for **local scratch** whose allocate/map calls may occur inside +conditional branches (`if (CS%store_CAu)`, `if (dyn_p_surf)` etc.) where the author was not +100%-confident every code path mapped the variable exactly once — `release`'s decrement-not-force +semantics degrade gracefully in that case. + +> **Resolved (2026-07-14):** The split is load-bearing, and in the dangerous direction — it is not a +> cosmetic convention. `exit data map(delete:)` forces the refcount to zero, so a per-call `delete` +> inside a callee destroys any outer, persistent mapping of the same object. This is live at HEAD: +> `vertvisc`'s (`MOM_vert_friction.F90`) per-call `map(delete: ADp)` kills `initialize_MOM`'s `ADp` +> map on the first call. Rule: `release` for scoped/per-call teardown; `delete` only in the owning +> `*_end` routine that mirrors the owning `enter data`. Never `map(delete:)` an object your scope +> does not own. + +--- + +## 4. No central mapping utility — hand-written, co-located with `ALLOC_`/`DEALLOC_` + +There is **no** wrapper macro, interface, or subroutine that performs "map this CS member" as a +single call. `MOM_memory_macros.h` defines only the *host*-side allocation macros +(`ALLOCABLE_`, `PTR_`, `ALLOC_(x)` → `allocate(x)`, `DEALLOC_(x)`, `TO_NULL_`); there is no +`MAP_ENTER_(x)`/`MAP_EXIT_(x)` counterpart anywhere in the tree (confirmed: no `omp` directives +appear inside `MOM_memory_macros.h`, and no `*.h`/module in `src/framework/` wraps a mapping +directive in a subroutine or macro — grep for `subroutine.*map\(` and `#define.*omp target` both +return nothing). + +Every `!$omp target enter/exit data`/`update` in the tree is a hand-written directive placed +immediately next to the corresponding `ALLOC_`/`DEALLOC_`/local-declaration line, as shown throughout +§1-§3. This is a **deliberate design choice**, not an oversight in progress: + +1. **Per-array map kind varies** (`to` vs `alloc` vs conditional — §3.5) based on whether the host + initializes the array before first device use, which a generic macro cannot infer without an + extra parameter that would have to be threaded through every call site anyway. +2. **Balance is visually auditable** only when the two directives sit next to their matching + `ALLOC_`/`DEALLOC_` — burying the mapping inside a macro/subroutine would hide exactly the + information (`15ca2a25f`'s missing `b_denom_1`) that a code reviewer needs to spot an imbalance. +3. **Struct-shell-before-members ordering (§2)** is call-site-specific — it depends on where in the + child module's own `_init` the member arrays get allocated, so a parent-side generic "map this CS" + helper would need the same insider knowledge a hand-written directive already encodes. +4. Bug history (`bc05a6a89`, `15ca2a25f`, `cdd3de9e3`→`1865612de`) shows the team iterating on + *placement and granularity* of directives per call site — premature abstraction into a shared + utility would have made these fixes harder, not easier, since each fix changed the *shape* of + what's being mapped (added a variable to a list, flattened a struct array, added an early-release + point), not a parameter to a generic call. + +Net effect: adding a GPU-resident array to any CS is a **copy-paste-and-adapt** operation from the +nearest analogous existing pattern in this document, not a call into shared infrastructure. + +--- + +## 5. `!$omp declare target` convention for device-callable helpers + +All 21 occurrences (see table in the summary) mark small, `pure`/`elemental`/side-effect-free +**column or point kernels** called from inside `do concurrent`/`omp target teams loop` regions: + +| File:line | Routine | Notes | +|---|---|---| +| `src/framework/MOM_coms.F90:69` | module data `pr, I_pr` (parameter arrays) | `!$omp declare target(pr, I_pr)` — the only *data* (not routine) declare-target in the inventory; these EFP precision-lookup tables must be resident for `efp_decompose` to read on device | +| `src/framework/MOM_coms.F90:779` | `efp_decompose` | `pure subroutine`; called per-real inside the block-reduction `do concurrent` of the reproducing sum (see `00-architecture.md` §7.2) | +| `src/framework/MOM_intrinsic_functions.F90:51` | `cuberoot` | `elemental function`; avoids `modulo()`/`pow()`-like intrinsics not implemented on all device targets | +| `src/framework/MOM_intrinsic_functions.F90:133,181,246` | `nth_root` family | bit-stable Newton iteration, explicitly documented (`:120-129`) as replacing `x**(1.0/n)` because device libm/libdevice differ in last-bit rounding from host | +| `src/parameterizations/vertical/MOM_vert_friction.F90:437,2101,2611` | `find_coupling_coef_gl90` and two others | column kernels called per-(i,j) inside the tridiagonal solve's `omp target teams loop collapse(2)` | +| `src/parameterizations/vertical/MOM_set_viscosity.F90:1251,1258,1294,1316,1389,1438,1718,1742,1802,1836,1966,2012` | `find_L_open_*` family (porous-topography open-fraction kernels) and BBL/ML column helpers | 12 of the 21 total — the single largest concentration; several routines carry the directive **twice** (once near the top of the subroutine body, once repeated just before the declarations end, e.g. `MOM_set_viscosity.F90:1251` and `:1258` inside the same `find_L_open_uniform_slope`) — harmless duplication, not two different routines | + +**The rule (from `00-architecture.md` §0.4/§7.5):** cross-module calls inside a device loop are +"painful" and must either be (a) inlined via `!NVF$ INLINE` / `-Minline=name:` compiler +flags, or (b) exposed as a `!$omp declare target` free routine with no polymorphic/`class(*)` +arguments and no unresolved external calls of its own — i.e. **declare target is necessary but not +sufficient; the routine must also actually inline or itself be fully declare-target reachable**. +This is why the EOS layer (`00-architecture.md` §7.1) could not simply add `!$omp declare target` to +its existing polymorphic dispatch — nvfortran cannot resolve the v-table on device regardless of the +directive, and duplicating each kernel as a `_loc` free function (`7c7af5572`) was required instead. + +--- + +## 6. `!$omp target update to/from(...)` — forced host↔device round-trips + +`update` is used, not to establish/tear down device residency (that's `enter`/`exit data`), but to +force a **fresh copy** in one direction while both host and device copies already exist. Three +recurring reasons appear in the inventory: + +### 6.1 Diagnostics posting (guarded by `id_* > 0`) + +The dominant use. `MOM_diag_mediator.F90`'s `post_data` is host-only and unchanged on `dev/gpu` +(`00-architecture.md` §7.4), so any field about to be posted must be pulled back first, and — since +that transfer is only needed if the diagnostic is actually requested this run — every such `update +from` is guarded by the diagnostic's registered ID being positive: + +```fortran +! src/tracer/MOM_tracer_hor_diff.F90:722 +!$omp target update from(khdt_x, khdt_y) if(CS%debug .or. CS%id_khdt_x>0 .or. CS%id_khdt_y>0) +!$omp target exit data map(release: khdt_x, khdt_y, Kh_u, Kh_v) map(release: CS) +... +if (CS%id_khdt_x > 0) call post_data(CS%id_khdt_x, khdt_x, CS%diag) +if (CS%id_khdt_y > 0) call post_data(CS%id_khdt_y, khdt_y, CS%diag) +``` + +and the Fortran-`if`-block variant, `MOM_tracer_hor_diff.F90:665-679`: + +```fortran +if (CS%id_KhTr_u > 0) then + !$omp target exit data map(from: Kh_u) + do j=js,je ; do I=is-1,ie + Kh_u(I,j,:) = G%mask2dCu(I,j)*Kh_u(I,j,1) ! host-side post-processing before posting + enddo ; enddo + ... + call post_data(CS%id_KhTr_u, Kh_u, CS%diag) +endif +``` + +Both forms exist: an OpenMP `if()` clause on `update` when the guard is a simple boolean +disjunction evaluated once, vs. a full Fortran `if` block (using `exit data map(from:)` rather than +`update from`) when the guarded region also does non-trivial host-side arithmetic before `post_data`. + +### 6.2 Host-only physics call bracketed by round-trips + +The split-RK2 driver (`MOM_dynamics_split_RK2.F90`) is riddled with `update to`/`update from` pairs +bracketing calls into modules/phases that are only partially ported, or bracketing debug-checksum +calls (§6.3) — e.g. around `PressureForce`/`CorAdCalc`/`vertvisc` transitions: + +```fortran +! :613-616 +!$omp target update from(CS%CAu_pred, CS%CAv_pred) +!$omp target update from(CS%PFu, CS%PFv, CS%pbce) +!$omp target update from(CS%diffu, CS%diffv) +!$omp target update from(u_bc_accel, v_bc_accel) +``` + +and the tracer underflow clean-up in `MOM_tracer_hor_diff.F90:1655-1660`, host-only scalar-threshold +logic wrapped in ordinary (unguarded) `update`: + +```fortran +if (Tr(m)%conc_underflow > 0.0) then + !$omp target update from(Tr(m)%t) + !$OMP parallel do default(shared) + do k=1,nz ; do j=js,je ; do i=is,ie + if (abs(Tr(m)%t(i,j,k)) < Tr(m)%conc_underflow) Tr(m)%t(i,j,k) = 0.0 + enddo ; enddo ; enddo + !$omp target update to(Tr(m)%t) +endif +``` + +### 6.3 Debug/checksum round-trips + +Guarded by `CS%debug`, e.g. the `CS%debug` disjunct in §6.1's `id_khdt_x` example, and generally +anywhere `MOM_state_chksum`/`uvchksum`/`hchksum` (`MOM_checksums.F90`, `00-architecture.md` §7.2) is +called on a field that lives on-device the rest of the time — these checksum calls are host-only, so +a `debug`-gated `update from` precedes them. + +### 6.4 Batch scalar refresh (`update to(CS)`) + +Whole-struct `update to(CS)`, distinct from the per-array round-trips above — see §2.2. Used after a +burst of host-side scalar/diag-ID computation on the CS, as a coarser-grained alternative to +enumerating each scalar member. + +--- + +## 7. Cross-references + +- `00-architecture.md` §2 (CS pattern, member styles), §6 (quantified inventory), §7.1 (EOS + polymorphism vs. declare target), §7.2 (EFP reproducing sums / `efp_decompose`), §7.3 (halos / + `omp_offload`), §7.4 (diagnostics still host-only), §7.5 (compiler workarounds). +- Key commits referenced here: `81680c15d` (CS/visc shell allocation, top-level partial-presence + fix), `1865612de` (struct-of-arrays → flat arrays, attach/detach cost), `bc05a6a89` (explicit + `h_tmp` map to avoid implicit transfers), `97629c240`/`cdd3de9e3` (incremental data-mapping + additions to `tracer_epipycnal_ML_diff`, later superseded in structure by `1865612de`), + `15ca2a25f` (missing `map(delete:)` balance bug), `7c7af5572` (EOS `_loc` free-function pattern, + referenced for the declare-target sufficiency rule in §5). + +--- + +## Verification notes + +Verified against source and git on branch `dev/gpu` (source + `git show`/`git log` only; no build/run). + +**Confirmed (spot-checked against the actual bytes):** + +- Directive counts across `src/ config_src/` reproduce exactly: **213** `enter data`, **168** + `exit data`, **398** `update`, **21** `declare target` (all 21 in `src/`; distribution + set_viscosity 12, vert_friction 3, intrinsic 4, coms 2 — sums to 21). The `168 (≈167)` hedge was + wrong-way-round (167 is the architecture doc's stale figure) and was corrected to `168`. +- All eight cited commit hashes resolve with the quoted subject lines (`81680c15d`, `1865612de`, + `bc05a6a89`, `15ca2a25f`, `cdd3de9e3`, `7c7af5572`, `97629c240`, `5b5f6b2b1`). +- §1.1/§1.2 lifecycle quotes are byte-exact at `MOM_dynamics_split_RK2.F90:1350-1368` + (`register_restarts_dyn_split_RK2`, 1329-1415) and `:2065-2083` (`end_dyn_split_RK2`). +- §1.3 local-scratch citations exact: `:435-436`, `:1193-1194`, `:1253`; continuity `:181`/`:228`, + `:658-660`. +- §1.4 `bc05a6a89` (h_tmp) and `15ca2a25f` (b_denom_1) diffs match the described before/after exactly. +- §2 shell-before-member sequence exact at `MOM_dynamics_split_RK2.F90:1689-1712`; `CS%pbv` block + exact at `MOM.F90:3225-3231`; `MOM_driver.F90:282`/`:636`; `MOM.F90:3277-3278` (`allocate(CS%visc)`). +- §2.2 `update to (CS)` at `MOM_barotropic.F90:6576`, `frhatu/frhatv`/`eta_cor` at `:6579-6580`, + `update to (CS%dtbt)` at `:6588`, and `MOM.F90:3104` all exact. +- §5 declare-target table: every line number verified (`MOM_coms.F90:69,779`; + `MOM_intrinsic_functions.F90:51,133,181,246`; `MOM_vert_friction.F90:437,2101,2611`; + `MOM_set_viscosity.F90:1251…2012` — 12 lines). +- §6 tracer_hor_diff quotes exact: `:665-680`, `:722-723`, `:1653-1662`. + +**Corrected:** + +1. **§3.3** — the `map(to: visc%Ray_v) if (allocated(visc%Ray_v))` example was cited at + `MOM_vert_friction.F90:437`, but `:437` is the `!$omp declare target` of `find_coupling_coef_gl90`, + and that `map` directive **no longer exists in current source** (it appears only in the diff + context of `15ca2a25f`; `Ray_v` is now read inline at `:942,954,1259,1267` and its scratch became + loop-`private`). Rewrote the bullet to make the live example the `Kv_shear` updates at `:1440-1441` + and flag the Ray_v form as historical. +2. **§3.4** — the `Reg%Tr(m)` snippet showed a single combined `map(to: …%t, …%df_x, …%df_y, …)`; + current source (`MOM_tracer_hor_diff.F90:209-216`) maps `%t` unconditionally and each optional + pointer field on its own `if(associated(...))`-guarded line. Replaced with the actual code. +3. **§1.4** — the balance-check example miscounted (`up, vp` "in three exit-data statements … two + enter-data statements"); actually entered once at `:436`, deleted in two statements (`:1194`, + `:1253`). Corrected the arithmetic and added why the double-delete is safe under nvfortran. + +**Enhanced:** added an attach/detach mechanism box in §2 (why shell-before-member, three mechanical +map-ordering rules, scalar members need no map); added a "critical copyback rule" to §3.5 +(`delete`/`release` never copy back — use `map(from:)`/`update from` first if the host needs the +value); sharpened the `delete`/`release` table rows to the correct force-to-zero vs decrement OpenMP +semantics; noted `allocated()` vs `associated()` guard selection for hybrid `vertvisc_type` fields. + +**Confidence:** High. Every file:line and every commit cited in the document was opened and matched; +the three corrections were the only substantive drifts (two stale-vs-current-code snippets and one +counting slip), and none affect the document's core patterns, which are all accurate. diff --git a/knowledge/gpu-knowledge/04-do-concurrent-patterns.md b/knowledge/gpu-knowledge/04-do-concurrent-patterns.md new file mode 100644 index 0000000..b5d29ea --- /dev/null +++ b/knowledge/gpu-knowledge/04-do-concurrent-patterns.md @@ -0,0 +1,611 @@ +# `do concurrent` Usage Patterns on `dev/gpu` + +> Companion to `00-architecture.md` §0(3), §5, §7.2. This document is about the *default parallel +> idiom* — Fortran `do concurrent` (DC) plus its F2018 locality specifiers — and the specific, +> catalogued cases where `!$omp target teams` is used instead. Repo: `dev/gpu`, baseline `dev-gfdl`. +> Source + git only; no build/run performed to produce this document. + +--- + +## 1. `DO_LOCALITY` and configure-time feature detection + +Not every Fortran compiler that MOM6 must build with supports F2018 locality specifiers on +`do concurrent` (`local`, `local_init`, `shared`, `default(none)`, `reduce`). To keep one source form +building everywhere, the port introduces a macro and an autoconf probe: + +- **Macro** — `src/framework/do_concurrent_compat.h`: + ```fortran + #ifndef DO_CONCURRENT_COMPAT_H_ + #define DO_CONCURRENT_COMPAT_H_ + #ifdef HAVE_FC_DO_CONCURRENT_LOCAL + #define DO_LOCALITY(X) X + #else + #define DO_LOCALITY(X) ; + #endif + #endif + ``` + When the compiler supports locality clauses, `DO_LOCALITY(local(k))` expands to `local(k)` (attached + to the `do concurrent` header). When it doesn't, it expands to `;` — a no-op statement separator — + so the loop compiles as a bare, unlocalized `do concurrent` (correct as long as the compiler treats + loop-body scalars conservatively/serially-consistent; on such compilers the code degrades toward + CPU-safe but does not GPU-parallelize the flagged locals — this is a portability fallback, not a + performance guarantee). + + > **Open (reviewed 2026-07-14):** What does a bare `do concurrent` with implicit (unspecified) + > locality actually generate — does it really degrade toward CPU-safe rather than GPU-parallelize the + > flagged locals? The review confirmed the caution stands but could not settle it from source; the + > decisive check is `-Minfo=accel` output on one kernel. See KNOWLEDGE.md §9. + +- **Feature probe** — `ac/m4/mom6_fc_do_concurrent_local.m4` (`MOM6_FC_DO_CONCURRENT_LOCAL`): + compiles a trivial `do concurrent(i=1:2) local(a,b)` program; if it compiles, + `mom6_cv_fc_do_concurrent_local=yes` and `AC_DEFINE([HAVE_FC_DO_CONCURRENT_LOCAL], [1], ...)`. + Invoked from `ac/configure.ac:173` (`MOM6_FC_DO_CONCURRENT_LOCAL`, under the `# Do concurrent + configuration` comment at `:172`), right after the real-8 flag setup (`:168`) and right before the + OpenMP configuration block (`:176`). + +- **Known gap** (m4 comment, verbatim): *"Currently only LOCAL is tested, but this should also + include LOCAL_INIT, SHARED, and DEFAULT(NONE)."* In practice the source already uses + `local_init` and `reduce` unconditionally gated by the same single `HAVE_FC_DO_CONCURRENT_LOCAL` + macro — i.e., the probe is a coarse yes/no gate for "any locality clause," not a per-clause matrix. + A compiler that supports `local` but not `reduce` (or vice versa) is not distinguished; this is a + latent risk flagged for future hardening, not yet hit in practice on the nvfortran target. + +**How the same source compiles everywhere:** every locality clause in the whole tree is written as +`DO_LOCALITY(...)`, never as a bare Fortran clause. A compiler lacking support only needs +`HAVE_FC_DO_CONCURRENT_LOCAL` left undefined by `configure`; no source edits, no `#ifdef` scattered +through physics code — the single macro in one header is the entire compatibility shim. + +--- + +## 2. Inventory: `do concurrent` and locality-clause usage across `src/` + +``` +grep -rn "do concurrent" src/ | wc -l → 698 +grep -rn "DO_LOCALITY" src/ | wc -l → 96 (94 real uses + 2 in the macro header itself: + the #define lines 7 and 9 of do_concurrent_compat.h) +``` + +### 2.1 Per-file `do concurrent` counts (all instances, with or without locality) + +| File | `do concurrent` count | `DO_LOCALITY` clause-lines | +|---|---:|:---:| +| `src/core/MOM_barotropic.F90` | 242 | 5 | +| `src/tracer/MOM_tracer_advect.F90` | 77 | 8 | +| `src/parameterizations/lateral/MOM_hor_visc.F90` | 62 | 0 | +| `src/core/MOM_continuity_PPM.F90` | 56 | 11 | +| `src/core/MOM_CoriolisAdv.F90` | 56 | 42 | +| `src/tracer/MOM_tracer_hor_diff.F90` | 32 | 9 | +| `src/core/MOM_PressureForce_FV.F90` | 30 | 0 | +| `src/parameterizations/vertical/MOM_set_viscosity.F90` | 28 | 6 | +| `src/parameterizations/vertical/MOM_vert_friction.F90` | 24 | 6 | +| `src/core/MOM_dynamics_split_RK2.F90` | 23 | 0 | +| `src/equation_of_state/MOM_EOS_Roquet_rho.F90` | 11 | 0 | +| `src/core/MOM_PressureForce_Montgomery.F90` | 10 | 0 | +| `src/equation_of_state/MOM_EOS_Wright.F90` | 9 | 0 | +| `src/diagnostics/MOM_sum_output.F90` | 9 | 4 | +| `src/core/MOM_interface_heights.F90` | 9 | 0 | +| `src/parameterizations/vertical/MOM_diabatic_aux.F90` | 7 | 0 | +| `src/core/MOM_forcing_type.F90` | 5 | 0 | +| `src/core/MOM.F90` | 5 | 0 | +| `src/framework/MOM_coms.F90` | (in `increment_block_ints`) | 3 (see §4.1) | + +(`DO_LOCALITY` clause-lines counts each `DO_LOCALITY(...)` occurrence textually — a single `do +concurrent` header split across a continuation line, e.g. `MOM_coms.F90:721-724`, contributes multiple +clause-lines for one loop; `MOM_CoriolisAdv.F90`'s 42 clause-lines cover 39 of its 56 `do concurrent` +loops, the remainder being continuation lines of an already-counted header.) + +**Important asymmetry:** most `do concurrent` loops (≈86%) carry **no** `DO_LOCALITY` at all — e.g. +`MOM_hor_visc.F90` (62), `MOM_PressureForce_FV.F90` (30), `MOM_dynamics_split_RK2.F90` (23) have zero. +This is not an oversight: a locality clause is only needed when the loop body declares/derives a +**scalar temporary that must be private per iteration** or performs a **reduction**. A very large +fraction of MOM6's `do concurrent` loops are pure elementwise array assignment +(`h(i,j,k) = max(hin(i,j,k) - dt*..., h_min)`, `MOM_continuity_PPM.F90:430`) with no loop-body scalar +state at all, so nothing needs declaring local. Examples with zero locals, no clause needed: +```fortran +! MOM_continuity_PPM.F90:429-431 — pure elementwise, no locality needed +do concurrent (k=1:nz, j=jsh:jeh, i=ish:ieh) + h(i,j,k) = max( hin(i,j,k) - dt * G%IareaT(i,j) * (uh(I,j,k) - uh(I-1,j,k)), h_min ) +enddo +``` +```fortran +! MOM_barotropic.F90:1087 — no locals, no clause +do concurrent (k=1:nz, j=js:je, I=is-1:ie) + ... +enddo +``` + +### 2.2 Categorized locality specifiers (all 94 real `DO_LOCALITY` uses) + +Counts below are **exact** textual `DO_LOCALITY(...)` occurrences (re-derived with per-clause greps, +e.g. `grep -rEn "DO_LOCALITY\(reduce\(\+" src/`); they sum to `67 + 2 + 5 + 11 + 9 = 94`, matching the +`96 − 2 header` total. The **mask** row is listed for contrast but is *not* a `DO_LOCALITY` clause, so +it does not count toward the 94. + +| Specifier | Count | Purpose | Representative file:line | +|---|---:|---|---| +| `local(...)` | 67 | Private per-iteration scalar/small-array temporaries (the dominant case) | `MOM_CoriolisAdv.F90:383` `local(k)`; `MOM_continuity_PPM.F90:2747` `local(h_im1,h_ip1)` | +| `local_init(...)` | 2 | Private per-iteration scalar that must **start** with its pre-loop value (conditionally overwritten inside the loop, then read unconditionally) | `MOM_set_viscosity.F90:682` `local_init(cdrag_sqrt_H, cdrag_sqrt_H_RL)`; `:759` `local_init(cdrag_sqrt_H)` | +| `reduce(+: ...)` | 5 | Integer/exact-arithmetic accumulation (reproducing sums, truncation counters) | `MOM_coms.F90:723` `reduce(+: block_sum)`; `MOM_vert_friction.F90:3189,3213,3259,3283` `reduce(+: ntrunc)` | +| `reduce(max: ...)` (10) + `reduce(min: ...)` (1) | 11 | Running max/min scalar (magnitude tracking, CFL/dt limits, "any work remaining" flags encoded as 0/1 max) | `MOM_coms.F90:724` `reduce(max: block_max_pos, block_max_neg, inan, iovf)`; `MOM_tracer_hor_diff.F90:919,963,969` `reduce(max: PEmax_kRho/itmp/k_size)`; `MOM_tracer_advect.F90:299,302,336,339,359,362` `reduce(max: domore_k_tmp)`; `MOM_barotropic.F90:3903` `reduce(min: min_max_dt2)` (the sole `reduce(min:)`) | +| `reduce(.or.: ...)` | 9 | Boolean "did anything happen" flags across a horizontal sweep | `MOM_vert_friction.F90:3168,3238` `reduce(.or.: trunc_any, do_any_write)`; `MOM_barotropic.F90:2984,3032,3053` `reduce(.or.: eta_is_submerged)`; `MOM_continuity_PPM.F90:870,1973` `reduce(.or.: any_simple_OBC)`; `MOM_tracer_advect.F90:582,1003` `reduce(.or.: domore_u_jk / domore_v_jk)` | +| Header **mask** expression (*not* a `DO_LOCALITY` clause — an F2008 scalar-logical restriction on the DC index set) | (not counted) | Skip iterations without branching in the body; also used to gate a subsequent write-back loop | `MOM_vert_friction.F90:3183` `do concurrent (j=js:je, I=Isq:Ieq, dowrite(I,j))`; `MOM_set_viscosity.F90:672,681` `do concurrent (i=is:ie, do_i(i,j))`; `MOM_tracer_hor_diff.F90:929,937` `..., G%mask2dT(i,j) > 0.0)` | + +**Confirmed: `shared(...)` and `default(none)` are never used on any `do concurrent` construct +anywhere in `src/`** (`grep -rn "do concurrent" src/ | grep -i "shared(\|default(none)"` → no matches). +Those clauses appear only on the **legacy CPU** directive `!$OMP parallel do default(shared)` / +`!$OMP parallel do default(none) shared(...)`, which persists in modules **not yet ported** to +`do concurrent` — `MOM_isopycnal_slopes.F90`, `MOM_dynamics_split_RK2b.F90` (the unsplit variant), +`MOM_kappa_shear.F90`, `MOM_set_diffusivity.F90`, `MOM_CVMix_KPP.F90`, `MOM_energetic_PBL.F90`, +`MOM_diabatic_driver.F90`, etc. — i.e. exactly the "untouched on `dev/gpu`" list in +`00-architecture.md` §4.3/§6.3. A few `!$OMP parallel do` survive even inside heavily-ported files +(e.g. `MOM_barotropic.F90:899-1060`, `MOM_set_viscosity.F90:2244-2548`) but only in cold/init-time or +alternate-configuration branches (e.g. the non-`linearized_BT_PV` else-branch at +`MOM_barotropic.F90:896-899`) that are not on the GPU-resident hot path — the two idioms coexist in +the same file without ever appearing on the same construct. + +--- + +## 3. Why `!$omp target teams` replaces `do concurrent` — two distinct categories + +`00-architecture.md` §0(3): *"`do concurrent` is the default parallel idiom. OpenMP `target teams` +directives are used only for reductions or where `do concurrent` misbehaves/underperforms."* The +`reduce()`-clause case is covered in §4 below (DC handles it directly once `HAVE_FC_DO_CONCURRENT_LOCAL` +is set — no `target teams` needed there). The two categories where the port falls back to explicit +OpenMP are: + +### 3.1 Category A — manual team count because the OpenMP runtime under-launched + +`MOM_continuity_PPM.F90:707` (`zonal_mass_flux`) and its meridional twin. The k-blocked kernel +processes one `niblock × njblock` tile per host-loop iteration; inside the tile, `!$omp target teams +num_teams(nteams)` wraps a serial `do k=1,nz` with `!$omp loop collapse(2) private(ii,jj)` per level, +calling `!$omp declare target` helpers `flux_elem` / `ratio_max`: + +```fortran +! MOM_continuity_PPM.F90:696-736 +do j_start=jsh,jeh,njblock ; do i_start=ish-1,ieh,niblock + i_end = min(i_start+niblock-1,ieh) + j_end = min(j_start+njblock-1,jeh) + + ! calculate number of teams + !$ nteams = ceiling(real((j_end-j_start+1)*(i_end-i_start+1))/128.) + ... + !$omp target teams num_teams(nteams) + do k=1,nz + ... + !$omp loop collapse(2) private(ii,jj) + do j=j_start,j_end ; do i=i_start,i_end + ii=i-i_start+1 ; jj=j-j_start+1 + call flux_elem(u(i,j,k),h_in(i,j,k),h_in(i+1,j,k),h_W(i,j,k),h_W(i+1,j,k),h_E(i,j,k),& + h_E(i+1,j,k),uh_t(ii,jj,k),duhdu(ii,jj,k),visc_rem(ii,jj,k),G%dy_Cu(i,j),& + G%IareaT(i,j),G%IareaT(i+1,j),G%IdxT(i,j),G%IdxT(i+1,j),dt,CS%vol_CFL,& + por_face_areaU(I,j,k)) + enddo ; enddo + ... + enddo + !$omp end target teams +``` + +**Why:** commit `5b5f6b2b1` ("add teams spec to problematic target region") — direct quote: +> *"For some reason omp runtime was only starting a kernel with 17 blocks when the openacc version +> would start it with 238 or something like that. Manually calculating number of teams sped it up."* + +I.e., nvfortran's default `omp target`/`omp target teams loop` team-count heuristic badly +under-subscribed the GPU for this particular tiled kernel shape (17 teams vs. the ~238 an OpenACC +`gang`-mapped equivalent got), so `nteams` is computed by hand +(`ceiling(real(tile_area)/128.)`) and pinned with `num_teams(nteams)` (occasionally paired with +`thread_limit(128)`, e.g. `MOM_set_viscosity.F90:803` +`!$omp target teams loop collapse(2) thread_limit(128)`). This whole family traces back to the port's +OpenACC→OpenMP migration, commit `3cb184edd` ("use openmp instead of openacc"): *"the translation +mapping from oacc to omp is an outer parallel region followed by multiple inner acc loops is +equivalent to an outer omp target followed by multiple inner omp loops... IMPORTANT: However for +OpenMP, inlining of ratio_max and flux_elem is MANDATORY... Otherwise results are incorrect."* Later, +`e8b0ecfbf` ("omp target teams loop -> do concurrent") reverted *other* regions of the same file back +to plain `do concurrent` once it was shown DC scheduled acceptably there — so `target teams` was not a +one-way migration; it is kept only where the manual-team-count fix is demonstrably needed +(`MOM_continuity_PPM.F90:707`), while nearby loops in the same file use bare +`do concurrent (k=1:nz, j=..., i=...)` (e.g. `:430`). + +### 3.2 Category B — `teams loop collapse(2)` wrapping a serial tridiagonal column solve + +`MOM_vert_friction.F90:737,938,1223,1255` (`teams loop collapse(2)`) and `:1443,1752` +(`teams distribute parallel do collapse(2)`); also `MOM_tracer_advect.F90:505,997,1169` and +`MOM_tracer_hor_diff.F90:1304,1465` (`teams loop` / `teams loop collapse(2)`). Representative kernel — +the u-momentum implicit vertical-friction tridiagonal solve: + +```fortran +! MOM_vert_friction.F90:735-786 (abridged) +!$omp target teams loop collapse(2) & +!$omp private(b1, c1, d1, Ray, b_denom_1) +do j=G%jsc,G%jec ; do I=Isq,Ieq ; if (G%mask2dCu(I,j) > 0.) then + ... + b1 = 1. / (b_denom_1 + dt * CS%a_u(I,j,2)) + d1 = b_denom_1 * b1 + u(I,j,1) = b1 * (CS%h_u(I,j,1) * u(I,j,1) + surface_stress(I,j)) + do k=2,nz ! <-- serial recurrence, NOT do concurrent + c1(k) = dt * CS%a_u(I,j,K) * b1 + b_denom_1 = CS%h_u(I,j,k) + dt * (Ray + CS%a_u(I,j,K) * d1) + b1 = 1. / (b_denom_1 + dt * CS%a_u(I,j,K+1)) + d1 = b_denom_1 * b1 + u(I,j,k) = (CS%h_u(I,j,k) * u(I,j,k) + dt * CS%a_u(I,j,K) * u(I,j,k-1)) * b1 + enddo + do k=nz-1,1,-1 ! <-- back-substitution, also serial + u(I,j,k) = u(I,j,k) + c1(k+1) * u(I,j,k+1) + enddo +endif ; enddo ; enddo +``` + +**Why not `do concurrent`:** the forward sweep (`b1`, `d1`, `c1(k)` each depend on the previous `k`'s +`b1`/`d1` — a genuine sequential recurrence, the Schopf & Loughe 1995 stable tridiagonal form) and the +back-substitution sweep are both inherently **serial in k**, per water column. Only the *(I,j)* +horizontal dimension is embarrassingly parallel (each column is independent). `do concurrent` has no +notion of "parallelize the outer two dimensions, run the third serially with private per-thread +scratch (`b1,c1,d1,Ray,b_denom_1`)" as a single construct — `collapse(2)` explicitly says which 2 of +the enclosing loops are the parallel ones, and `private(...)` gives each (I,j) team/thread its own +scratch for the serial k-recurrence it runs internally. This is the exact idiom flagged in +`00-architecture.md` §4.3: *"Uses `!$omp target teams loop collapse(2)` with a serial inner +tridiagonal k-loop."* The same shape recurs in tracer advection/diffusion wherever a column-local +sequential dependency (limiter passes, vertical remap-like bookkeeping) sits inside an otherwise +horizontally-parallel loop. + +### 3.3 The traffic is bidirectional, not a one-way port + +Commits `0631c70bc`, `b4ae33d5c`, `e8b0ecfbf` are all literally titled **"omp target teams loop -> +do concurrent"** — i.e. `target teams loop` was tried first (straight OpenACC-style translation), +then *reverted to* `do concurrent` once it was shown to compile/perform acceptably as DC. Conversely +`5b5f6b2b1`, `dbc2521d1`, `0d2f4d7d7`, `38fd0ae32` are all titled **"add teams spec to problematic +target region"** — the opposite direction, applied surgically to the handful of regions where DC (or +plain `omp target`) demonstrably mis-scheduled. `b95083139` ("include k-loop in do concurrent") shows +a third motion entirely within the DC world: merging a separate per-k kernel launch into the DC header +itself (§5 below) for a 20%-ish speedup by cutting launch count — orthogonal to the DC-vs-teams +question, but part of the same tuning cycle. + +--- + +## 4. Reduction idioms — why flags/reductions instead of writing CS/module globals from a loop + +A `do concurrent`/GPU kernel body must not write a shared scalar (a `CS%` member, a module variable) +directly from every iteration — that is either a data race (undefined result under concurrent +execution) or, if serialized, defeats parallelism. The port's answer is always: **reduce into a local +scalar, then commit the scalar to persistent state once, after the loop, on the host/serial side.** + +### 4.1 Exact-integer sums for reproducibility (`MOM_coms.F90`) + +`increment_block_ints` (`MOM_coms.F90:695-772`), the core of the block-based reproducing sum +(`00-architecture.md` §7.2): + +```fortran +! MOM_coms.F90:721-724, 741 +do concurrent (j=jbs:jbe, i=ibs:ibe) & + DO_LOCALITY(local(r, e, rmag, lnan, lovf)) & + DO_LOCALITY(reduce(+: block_sum)) & + DO_LOCALITY(reduce(max: block_max_pos, block_max_neg, inan, iovf)) + r = descale * array(i,j) + call efp_decompose(r, e, rmag, lnan, lovf) + inan = max(inan, lnan) ; iovf = max(iovf, lovf) + if (r >= 0.) then ; if (rmag > block_max_pos) block_max_pos = rmag + else ; if (rmag > block_max_neg) block_max_neg = rmag ; endif + block_sum(:) = block_sum(:) + e(:) ! reduce(+: block_sum) — exact fixed-point carry array +enddo +``` +`e(:)`/`block_sum(:)` are the fixed-point EFP carry-limbs (`00-architecture.md` §7.2); reducing over +*exact integers* (not floats) means the result is bit-identical regardless of thread/team scheduling +order — this is precisely what buys bitwise reproducibility on GPU. `inan`/`iovf` are `reduce(max:)` +"did we hit a NaN/overflow anywhere" flags, converted to `NaN_error`/`overflow_error` module state +*after* the loop (`:770-771`) — never written from inside the loop. + +### 4.2 `ntrunc` truncation counters and `.or.` flags (`MOM_vert_friction.F90:3167-3283`) + +```fortran +! MOM_vert_friction.F90:3162-3200 (u-component; v-component is the mirror at :3232-3271) +do concurrent (j=js:je, I=Isq:Ieq) + dowrite(I,j) = .false. ; vel_report(I,j) = 3.0e8 * US%m_s_to_L_T +enddo + +do concurrent (k=1:nz, j=js:je, I=Isq:Ieq) & + DO_LOCALITY(reduce(.or.: trunc_any, do_any_write)) + ... + if (CFL > CS%CFL_trunc) trunc_any = .true. + if (CFL > CS%CFL_report) then + dowrite(I,j) = .true. ; do_any_write = .true. + vel_report(I,j) = min(vel_report(I,j), abs(u(I,j,k))) + endif +enddo + +do concurrent (j=js:je, I=Isq:Ieq, dowrite(I,j)) ! <-- mask specifier, not a locality clause + u_old(I,j,:) = u(I,j,:) +enddo + +if (trunc_any) then + ntrunc = 0 + do concurrent (k=1:nz, j=js:je, I=Isq:Ieq) DO_LOCALITY(reduce(+: ntrunc)) + ... if (...) ntrunc = ntrunc + 1 + enddo + CS%ntrunc = CS%ntrunc + ntrunc ! <-- CS state updated once, outside/after the DC loop +endif +``` +Two textbook reasons flags/reductions are used instead of touching `CS%` fields in-loop: +1. **`CS%ntrunc` is persistent cross-timestep state.** Incrementing it from inside a concurrent loop + would race; instead every iteration increments a *local* `ntrunc`, and `CS%ntrunc = CS%ntrunc + + ntrunc` happens exactly once, serially, after the reduction completes. +2. **Control flow decisions** (`if (trunc_any) then ...`, `if (do_any_write) then ... call + write_u_accel(...)`) **must be made on the host/serial side** — you cannot conditionally branch + per-GPU-iteration into a diagnostic I/O call (`write_u_accel`); `trunc_any`/`do_any_write` are + `reduce(.or.:)` booleans precisely so the *aggregate* "did any point trip" question can be answered + once, then used to gate a genuinely serial follow-up (`!$omp target update from(u_old, + vel_report)` + a plain host `do j=... ; do I=... ; if (dowrite(I,j)) call write_u_accel(...)`, + `:3204-3209`). The mask `dowrite(I,j)` at `:3183` is the vehicle that carries the per-point decision + from the reduction pass to the later write-back pass without re-branching inside a device loop. + +### 4.3 `domore_*` flags replacing early-exit search (`MOM_tracer_advect.F90`) + +```fortran +! MOM_tracer_advect.F90:296-306 +domore_k_tmp = 0 +do concurrent (j=jsv:jev, domore_u(j,k)) DO_LOCALITY(reduce(max:domore_k_tmp)) + domore_k_tmp = 1 +enddo +do concurrent (J=jsv+stencil-1:jev-stencil, domore_v(J,k)) DO_LOCALITY(reduce(max:domore_k_tmp)) + domore_k_tmp = 1 +enddo +domore_k(k) = domore_k_tmp +``` +and (`:582`) `do concurrent (I=is-1:ie) DO_LOCALITY(reduce(.or.:domore_u_jk))`. These replace a +sequential "scan until you find one true value, then stop" idiom (illegal to parallelize directly) +with a `reduce(max:)`/`reduce(.or.:)` over a 0/1 or logical flag set per iteration — equivalent to a +data-parallel logical-OR — used to synchronize the multi-pass mass-flux-limiting iteration +(`domore_k` gates whether level `k` needs another advection pass) without any per-iteration branch +into shared state. Note line 284-294 nests an ordinary serial `do i ... ; exit` search *inside* the +body of an outer `do concurrent (j=..., domore_k(k)>0)` — legal, because `exit` only terminates the +inner sequential `do`, never crosses a `do concurrent` boundary. + +### 4.4 A genuine nvfortran limitation: reductions must target scalars, not array elements + +`MOM_tracer_hor_diff.F90:959-971`, with an explicit in-source comment: +```fortran +! MOM_tracer_hor_diff.F90:959-971 +do concurrent (j=js-1:je+1) DO_LOCALITY(local(itmp)) + itmp = 0 + ! nvfortran do concurrent cannot reduce array elements + do concurrent (i=is-1:ie+1) DO_LOCALITY(reduce(max:itmp)) + itmp = max(itmp, num_srt(i,j)) + enddo + max_srt(j) = itmp +enddo +k_size = 1 +do concurrent (j=js-1:je+1) DO_LOCALITY(reduce(max:k_size)) + k_size = max(k_size, 2*max_srt(j)) +enddo +``` +The natural write would be `reduce(max: max_srt(j))` inside the inner loop, but nvfortran rejects (or +mishandles) reducing into an indexed array element — only a bare scalar can be a `reduce()` target. The +workaround stages the per-`j` maximum into a scalar `itmp` (privatized per outer `j` iteration via +`local(itmp)`), reduces into `itmp` over the inner `i` loop, then does a plain (non-reducing) scalar +assignment `max_srt(j) = itmp` in the outer iteration. This is the same reason `domore_k_tmp` (§4.3) +and `block_sum`/`block_max_pos` (§4.1, which *is* allowed to be a whole-array reduce target — arrays +are fine as long as they aren't *indexed elements* being reduced individually) exist as standalone +scalars/arrays rather than being reduced straight into `CS%`-member array elements. + +> **Resolved (2026-07-14):** Rejecting an array-element `reduce` is conforming F2023, not an nvfortran +> quirk. A locality-spec/`reduce` list takes *variable names*; `max_srt(j)` is an array element, not a +> variable, so no conforming compiler accepts it. Whole-array `reduce(+: block_sum)` is conforming and +> positively supported — "whole-array reduce is fine" is a portable rule, not a local observation. The +> staged-scalar workaround above stays correct. + +--- + +## 5. Loop-nest order and where k stays serial vs. joins the `do concurrent` header + +### 5.1 Index-order census + +Arity is counted by the number of `=` signs in each header (each `v=range` has exactly one; masks +carry none, and no header uses `>=`/`<=`/`==`, so `#(=)` is exactly the index count). Of the 698 +`do concurrent`, 685 have a single-line header captured this way; the remaining ~13 span a +continuation line: + +``` +1-index headers: 126 — leading: i 58, j 26, I 14, J 14, m 9, k 2 + (row sweeps at fixed j,k; (m=1:ntr) tracer loops; a few (k)/(i)) +2-index headers: 390 — leading: j 218, J 129, k 27, jj 10, m 3 + (dominated by (j,i)/(J,I) horizontal pairs) +3-index headers: 169 — leading: k 114, kk 54, j 1 + (kk = block-local level index, k re-derived in the body — see §5.2) +``` +**Convention (verified, 0 counterexamples):** whenever `k` (or `kk`) appears in a `do concurrent` +header at all, it is written **first** — `(k=1:nz, j=..., i=...)` / `(kk=1:kmax, J=..., I=...)`, never +last (`grep` for a header with `k=`/`kk=` in any non-leading position returns **nothing**). Among the +169 three-index headers, 168 lead with `k`/`kk` and only one leads with `j`. Horizontal-only 2D sweeps +(the plurality of all DC loops, 390) use `(j,i)`/`(J,I)` order, center-point before corner-point naming +following the grid convention in `00-architecture.md` §3.3. All 54 `kk`-led headers live in +`MOM_CoriolisAdv.F90` (its k-blocked kernels). + +### 5.2 k inside the `do concurrent` header — the k-blocking connection + +When a subroutine is k-blocked (`00-architecture.md` §5), the header index is a **block-local** `kk` +running `1:kmax` (where `kmax = k_end - k_start + 1`), and the **absolute** level `k` is a derived +scalar computed on the first line of the loop body and then used for all array indexing that must +reference the global level: + +```fortran +! MOM_CoriolisAdv.F90:383-389 — kk drives the DC header, k is a body-local derived scalar +do concurrent (kk=1:kmax, J=Js_q:Je_q, I=Is_q:Ie_q) DO_LOCALITY(local(k)) + k = k_start + kk - 1 + dvSdx(I,J,kk) = (-Waves%us_y(i+1,J,k)*G%dyCv(i+1,J)) - (-Waves%us_y(i,J,k)*G%dyCv(i,J)) + duSdy(I,J,kk) = (-Waves%us_x(I,j+1,k)*G%dxCu(I,j+1)) - (-Waves%us_x(I,j,k)*G%dxCu(I,j)) +enddo +``` +`local(k)` is mandatory here: without it, `k` would be treated as a single shared variable written by +every concurrent iteration — a race, and wrong on every iteration but the "last" one under any +serialized interpretation. Declaring it `local` gives each `(kk,J,I)` iteration its own private copy, +which is exactly what makes the k-blocking transformation (whole-domain block on GPU, small cache +block on CPU) safe to express as a single `do concurrent` regardless of block size. This is far and +away the dominant reason `local(k)` appears (`MOM_CoriolisAdv.F90` alone: 39 of its 56 `do concurrent` +loops carry a `DO_LOCALITY(local(k...` clause). + +### 5.3 k as a genuinely serial inner loop (not in the DC header at all) + +Two distinct reasons k is pulled *out* of the `do concurrent` header and left as an ordinary serial +`do k=...` nested inside the parallel horizontal loop: + +1. **Sequential recurrence** (§3.2) — tridiagonal forward/back-substitution in + `MOM_vert_friction.F90:752-769,776-785`: `do concurrent`/`teams loop collapse(2)` over `(I,j)` + (independent columns), ordinary serial `do k=2,nz` / `do k=nz-1,1,-1` inside each column body, + because `b1`/`d1`/`u(...,k)` at level `k` depend on level `k-1`'s just-computed values. +2. **Historical/simplicity, later collapsed for performance** — `MOM_set_viscosity.F90:697-724`, a + `do k=nz,1,-1 ; if (htot_vel>=CS%Hbbl) exit` bottom-boundary-layer accumulation with data-dependent + early termination (`exit`) — inherently serial per column regardless of GPU target, nested inside + an outer `do concurrent (i=is:ie, do_i(i,j)) DO_LOCALITY(local(k, cdrag_sqrt)) + DO_LOCALITY(local_init(cdrag_sqrt_H, cdrag_sqrt_H_RL))`. Commit `b95083139` ("include k-loop in do + concurrent") documents the *opposite* move for a different case in the same file/module family — + originally `MOM_vert_friction.F90`'s velocity-truncation logic issued **one kernel launch per k** + (a host `do k=1,nz` wrapping single-level 2D `do concurrent` calls); the commit message: *"the + velocity truncation is doing k launches of ij-kernels to truncate velocity. There's no real reason + for the k loop to be separate from the ij loops, so merging them... for 500x500x100 grid, speeds up + total vertvisc time by 20-ish%."* That merge produced exactly the `do concurrent (k=1:nz, j=js:je, + I=Isq:Ieq) DO_LOCALITY(reduce(+: ntrunc))` form quoted in §4.2 — k joins the header whenever there + is **no** cross-k dependency, purely to amortize kernel-launch overhead; it stays a serial inner + loop only when correctness (recurrence, data-dependent early exit) requires it. + +### 5.4 Rule of thumb + +| Situation | Loop form | +|---|---| +| Elementwise/independent-per-level physics, no recurrence | `k` (or blocked `kk`) folded into the `do concurrent` header, `local(k)` if `k` is a derived scalar | +| Column has a top-to-bottom (or bottom-to-top) sequential dependency (tridiagonal solve, running accumulation with early exit) | `k` left as an ordinary serial `do`, nested inside a DC/`teams loop collapse(2)` over the horizontal indices only, with `private`/`local` scratch for the per-column recurrence state | +| A scalar must be aggregated across the whole iteration space (sum/max/min/or) | `reduce(...)` clause on whichever loop performs the aggregation; never accumulate directly into `CS%`/module state inside the loop | + +### 5.5 Prescriptive decision tree — "which loop form do I write?" + +Walk these branches **top to bottom** and take the first that matches. Every branch is grounded in a +verified in-tree example; copy that example's shape. + +1. **Is the loop body a pure elementwise / independent-per-point assignment** — every RHS reads only + its own `(i,j,k)` (and neighbours), every LHS is a distinct array element, no loop-body scalar is + carried and nothing is aggregated? + → **Bare `do concurrent`, no clause.** Fold `k`/`kk` into the header first. + *Pattern:* `MOM_continuity_PPM.F90:430` + `do concurrent (k=1:nz, j=jsh:jeh, i=ish:ieh)`; also `MOM_barotropic.F90:1087`. This is ≈86% of all + DC loops — do **not** reach for a clause you don't need. + +2. **Does the body compute one or more scalar (or tiny fixed-size) temporaries that must be private + per iteration** — a re-derived index, a `sqrt`, a reused neighbour value — and each is *written + before it is read* within the same iteration? + → **`DO_LOCALITY(local(...))`.** + *Pattern:* `MOM_CoriolisAdv.F90:383` `do concurrent (kk=1:kmax, J=…, I=…) DO_LOCALITY(local(k))` + with `k = k_start + kk - 1` on the first body line (the k-blocking idiom, §5.2); or + `MOM_continuity_PPM.F90:2747` `DO_LOCALITY(local(h_im1,h_ip1))`. Without `local`, the scalar is a + shared write = race. + +3. **Same as (2), but the private scalar must *start* each iteration holding its pre-loop value** + because the loop only *conditionally* overwrites it and then reads it unconditionally? + → **`DO_LOCALITY(local_init(...))`** (add a plain `local(...)` for the always-written temps in the + same header). + *Pattern:* `MOM_set_viscosity.F90:681-682` + `do concurrent (i=is:ie, do_i(i,j)) DO_LOCALITY(local(k, cdrag_sqrt)) DO_LOCALITY(local_init(cdrag_sqrt_H, cdrag_sqrt_H_RL))` + — `cdrag_sqrt_H` is set only inside `if (CS%bottomdragmap)`, so `local_init` carries the outer value + for the else-path. Only **2** such loops exist tree-wide; use it only when the conditional-init + pattern genuinely holds. + +4. **Do you need to aggregate one value across the whole iteration space** (sum, max, min, logical + or)? + - **Target is a bare scalar (or a whole array reduced as a unit):** put a + **`DO_LOCALITY(reduce(: ))`** on the loop; commit it to `CS%`/module state **once, + after** the loop. + *Pattern:* `MOM_vert_friction.F90:3189` `… DO_LOCALITY(reduce(+: ntrunc))` then + `CS%ntrunc = CS%ntrunc + ntrunc` (§4.2); exact-integer `reduce(+: block_sum)` + + `reduce(max: …)` at `MOM_coms.F90:723-724` (§4.1); boolean gate + `reduce(.or.: trunc_any, do_any_write)` at `:3168`. + - **Target is an *indexed array element* `a(j)`:** nvfortran rejects it. **Stage a scalar** — + `local(itmp)` on the outer loop, `reduce(: itmp)` on the inner loop, then plain + `a(j) = itmp`. + *Pattern:* `MOM_tracer_hor_diff.F90:959-966` (see the in-source comment at `:962`, §4.4). + - **A `reduce()` you expected to work is rejected / mis-scheduled, or the aggregation sits over + independent columns you also want teamed:** fall back to explicit OpenMP (next branches). + +5. **Is there a genuine sequential recurrence down a column** — `x(k)` depends on `x(k-1)` (tridiagonal + forward/back-substitution, running accumulation)? + → **`!$omp target teams loop collapse(2)` over the horizontal `(I,j)` only, with `private(...)` for + the per-column scratch, and a plain serial `do k=…` inside.** `do concurrent` cannot express + "parallelize 2 dims, run the 3rd serially with private scratch" as one construct. + *Pattern:* `MOM_vert_friction.F90:737-786` + `!$omp target teams loop collapse(2) private(b1, c1, d1, Ray, b_denom_1)` wrapping serial + `do k=2,nz` / `do k=nz-1,1,-1` (§3.2). The same shape recurs at `:938,1223,1255` and in tracer + advect/diff limiter passes (`MOM_tracer_advect.F90:505,997,1169`; + `MOM_tracer_hor_diff.F90:1304,1465`). + *Contrast:* a column loop with only a **data-dependent early `exit`** (not a recurrence) can stay a + serial `do k=…` nested inside a *`do concurrent`* over the horizontal — `MOM_set_viscosity.F90:697` + (`do k=nz,1,-1 ; if (htot_vel>=CS%Hbbl) exit`) — no OpenMP needed. + +6. **You wrote a plain `do concurrent` / `omp target teams loop` and profiling shows the GPU is badly + under-subscribed** (few teams launched for a large tile)? + → **Compute the team count by hand and pin it: `!$omp target teams num_teams(nteams)`** (optionally + `thread_limit(128)`), with the compute split into `!$omp loop collapse(2)` regions calling + `!$omp declare target` helpers (which **must** inline — `-Minline` / `!NVF$ INLINE`). + *Pattern:* `MOM_continuity_PPM.F90:696-736`, + `nteams = ceiling(real(tile_area)/128.)` then `!$omp target teams num_teams(nteams)` at `:707` + (commit `5b5f6b2b1`: 17 teams → 238); `thread_limit(128)` at `MOM_set_viscosity.F90:803`. Apply this + **surgically** — it is the exception, not the default. Conversely, if a hand-placed + `target teams loop` schedules fine as plain DC, revert it (commits `e8b0ecfbf`, `0631c70bc`, + `b4ae33d5c`, all "omp target teams loop -> do concurrent"). + +**Default bias:** branches 1-4 (`do concurrent`, with a clause only when forced) cover the +overwhelming majority; branches 5-6 (explicit OpenMP) are the two catalogued, evidence-backed +exceptions. Reach for OpenMP only when a column recurrence (5) or a measured under-launch (6) makes DC +insufficient. + +--- + +## 6. Cross-references + +- `docs/gpu-knowledge/00-architecture.md` §0(3) guiding principle, §5 k-blocking, §6.1 directive + totals (698/829/213/167/21), §7.2 EFP reproducing sums, §9 quick-reference recipe. +- Compiler-workaround commits referenced here: `5b5f6b2b1`, `3cb184edd`, `e8b0ecfbf`, `0631c70bc`, + `b4ae33d5c`, `dbc2521d1`, `0d2f4d7d7`, `38fd0ae32`, `b95083139`. +- Infra: `src/framework/do_concurrent_compat.h`, `ac/m4/mom6_fc_do_concurrent_local.m4`, + `ac/configure.ac:173`. + +--- + +## Verification notes + +Independent Opus verification pass (source + git only; no build/run). Every count re-derived with +fresh greps; every cited exemplar line re-read; every quoted commit body re-fetched. + +**Confirmed (unchanged):** +- Totals `698 do concurrent`, `96 DO_LOCALITY`, and **all 19 per-file counts** in the §2.1 table + (barotropic 242/5, tracer_advect 77/8, hor_visc 62/0, continuity 56/11, CoriolisAdv 56/42, … coms + 1/3) reproduce exactly. +- `do_concurrent_compat.h` macro and `mom6_fc_do_concurrent_local.m4` probe (incl. the "Currently only + LOCAL is tested…" comment) verbatim as quoted. `shared(...)`/`default(none)` never appear on any DC + construct (0 matches). +- `local_init` at `MOM_set_viscosity.F90:682,759` and its conditional-init rationale; the mask idiom + and reductions at `MOM_vert_friction.F90:3162-3209` (`:3183` mask, `:3189` `reduce(+: ntrunc)`); + the array-element-reduce comment at `MOM_tracer_hor_diff.F90:962`; the `num_teams` kernel at + `MOM_continuity_PPM.F90:696-736` (`:707`); the tridiagonal `teams loop collapse(2)` at + `MOM_vert_friction.F90:737-786`; the `kk`/`k` re-derivation at `MOM_CoriolisAdv.F90:383-384` + (`local(k)` = 39 of its loops) — all confirmed. +- Commit subjects **and bodies** verbatim: `5b5f6b2b1` ("17 blocks … 238"), `3cb184edd` + (OpenACC→OpenMP, inlining mandatory), `b95083139` (20%-ish), and the `e8b0ecfbf`/`0631c70bc`/ + `b4ae33d5c` reverts / `dbc2521d1`/`0d2f4d7d7`/`38fd0ae32` teams-spec additions. + +**Corrected:** +- `ac/configure.ac` line: the `MOM6_FC_DO_CONCURRENT_LOCAL` invocation is at **`:173`** (comment at + `:172`), not `:172` (two occurrences fixed). *(Note: `00-architecture.md` §6.1 still says `:172` and + is out of this doc's edit scope.)* +- `DO_LOCALITY` breakdown: **94** real uses + **2** in the header (not "95 + 1"); the header defines + the macro on two `#define` lines (7 and 9). +- §2.2 category counts made exact: `local` **67** (was ~70), `reduce(+)` **5** (was 4), `reduce(max)` + **10** + `reduce(min)` **1** = **11** (was ~13), `reduce(.or.)` **9** (was ~7); these sum to + `67+2+5+11+9 = 94`. `local_init` = 2 was already correct. +- §5.1 index-order census rewritten: it was internally inconsistent (a "3-index = 165 total" claim + whose own leading-index rows summed to 217, having conflated 3-index leading with all-header + leading). Corrected via `=`-count arity: **1-index 126, 2-index 390, 3-index 169**; three-index + leading is **k 114, kk 54, j 1** (168 of 169 lead with k/kk; 0 headers put k/kk non-leading). The + `kk`=54 figure survives and all 54 are in `MOM_CoriolisAdv.F90`. + +**Enhancements:** added §5.5, a 6-branch prescriptive "which loop form" decision tree (bare DC → +`local` → `local_init` → `reduce`/staged-scalar/OpenMP → `teams loop collapse(2)` for k-recurrence → +manual `num_teams` for under-launch), each branch grounded in a verified file:line + commit. + +**Confidence:** High. Every numeric claim was recomputed and every exemplar/commit re-read against the +`dev/gpu` tree; the only residual uncertainty is the implicit-locality codegen question flagged in §1, +which needs `-Minfo=accel` output rather than more source reading. diff --git a/knowledge/gpu-knowledge/05-kblocking-tiling.md b/knowledge/gpu-knowledge/05-kblocking-tiling.md new file mode 100644 index 0000000..7262838 --- /dev/null +++ b/knowledge/gpu-knowledge/05-kblocking-tiling.md @@ -0,0 +1,859 @@ +# K-blocking / Tiling — the Blessed CPU-and-GPU-Preserving Refactor + +> Companion to `00-architecture.md` §5 (its one-paragraph summary is the seed for this document). +> This document reconstructs the *exact* mechanical transformation from git history across three +> cases — continuity (`MOM_continuity_PPM.F90`, fully merged, i/j **and** k blocking), CoriolisAdv +> (`MOM_CoriolisAdv.F90`, merged `b8c471cfa`, k-blocking only), and horizontal viscosity +> (`MOM_hor_visc.F90`, in-flight on `kblock-hor-visc`, k-blocking only) — and distills a template. +> Read `00-architecture.md` first. + +--- + +## 1. The mechanical transformation, step by step + +There are two distinct blocking axes in this codebase, and it is important not to conflate them: + +- **Horizontal (i/j) tiling** — only in `MOM_continuity_PPM.F90`. A 2-D (or implicitly 3-D, looping + `k` innermost/serially) computation is chopped into rectangular `(niblock × njblock)` tiles; the + host loops over tiles, and each tile's body becomes a small, separately-launched device kernel. +- **Vertical (k) blocking** — in all three modules. A `do k=1,nz` **serial outer loop** whose body + is a 2-D horizontal calculation (with 2-D scratch/work arrays reused every iteration) is turned + into a loop over **blocks of `nkblock` adjacent layers**, so that up to `nkblock` layers can be + exposed simultaneously to `do concurrent`/`omp target`. Scratch arrays grow one extra dimension of + size `nkblock` (not `nz`) so device memory footprint stays bounded regardless of column depth. + +Both axes follow the same recipe; k-blocking is the more general and more widely used one, so it is +the primary subject below. + +### 1.1 Vertical (k) blocking — before/after (continuity reconstruction) + +This is commit `93dbbd36e` ("Use blocking in k dimension for continuity reconstruction (#165)"), +`src/core/MOM_continuity_PPM.F90`, subroutine `PPM_reconstruction_x`. Before: + +```fortran +! integer :: k ! vertical grid index +real, dimension(SZI_(G),SZJ_(G),SZK_(GV)) :: slp ! full-column scratch slope array +... +do concurrent (k=1:nz, j=jsl:jel, i=isl-1:iel+1) + ... + slp(i,j,k) = sign(1.,slp(i,j,k)) * min(abs(slp(i,j,k)), 2. * min(dMx, dMn)) +enddo +... +do concurrent (k=1:nz, j=jsl:jel, i=isl:iel) + h_im1 = G%mask2dT(i-1,j) * h_in(i-1,j,k) + (1.0-G%mask2dT(i-1,j)) * h_in(i,j,k) + h_ip1 = G%mask2dT(i+1,j) * h_in(i+1,j,k) + (1.0-G%mask2dT(i+1,j)) * h_in(i,j,k) + h_W(i,j,k) = 0.5*( h_im1 + h_in(i,j,k) ) + oneSixth*( slp(i-1,j,k) - slp(i,j,k) ) + h_E(i,j,k) = 0.5*( h_ip1 + h_in(i,j,k) ) + oneSixth*( slp(i,j,k) - slp(i+1,j,k) ) +enddo +if (monotonic) then + call PPM_limit_CW84(h_in, h_W, h_E, G, GV, isl, iel, jsl, jel, nz) +else + call PPM_limit_pos(h_in, h_W, h_E, h_min, G, GV, isl, iel, jsl, jel, nz) +endif +``` + +After (`src/core/MOM_continuity_PPM.F90:2703-2832` today): + +```fortran +integer :: k, kk ! vertical grid and k-block index +real, dimension(SZI_(G),SZJ_(G),max(1,nkblock)) :: slp ! one k-BLOCK of scratch, not the full column +... +do ks = 1, nz, nkblock + ke = min(ks + nkblock - 1, nz) + ... + do concurrent (k=ks:ke, j=jsl:jel, i=isl-1:iel+1) DO_LOCALITY(local(dMx,dMn,kk)) + kk = k - ks + 1 + slp(i,j,kk) = sign(1.,slp(i,j,kk)) * min(abs(slp(i,j,kk)), 2. * min(dMx, dMn)) + enddo + ... + do concurrent (k=ks:ke, j=jsl:jel, i=isl:iel) DO_LOCALITY(local(h_im1,h_ip1,kk)) + kk = k - ks + 1 + h_im1 = G%mask2dT(i-1,j) * h_in(i-1,j,k) + (1.0-G%mask2dT(i-1,j)) * h_in(i,j,k) + h_ip1 = G%mask2dT(i+1,j) * h_in(i+1,j,k) + (1.0-G%mask2dT(i+1,j)) * h_in(i,j,k) + h_W(i,j,k) = 0.5*( h_im1 + h_in(i,j,k) ) + oneSixth*( slp(i-1,j,kk) - slp(i,j,kk) ) + h_E(i,j,k) = 0.5*( h_ip1 + h_in(i,j,k) ) + oneSixth*( slp(i,j,kk) - slp(i+1,j,kk) ) + enddo + if (monotonic) then + call PPM_limit_CW84(h_in, h_W, h_E, G, GV, isl, iel, jsl, jel, ks, ke) + else + call PPM_limit_pos(h_in, h_W, h_E, h_min, G, GV, isl, iel, jsl, jel, ks, ke) + endif +enddo +``` + +The five-part mechanical recipe visible in this diff (repeated verbatim in CoriolisAdv and +hor_visc): + +1. **Add a block-size CS member** (`nkblock`, or `niblock`/`njblock`) and thread it into the + subroutine as an argument (`MOM_continuity_PPM.F90:76-78`, `2693`). +2. **Wrap the serial/parallel `k=1:nz` axis in an outer host loop over block starts**: + `do ks = 1, nz, nkblock ; ke = min(ks+nkblock-1, nz)`. For i/j tiling the equivalent is + `do j_start=jsh,jeh,njblock ; do i_start=ish-1,ieh,niblock` (`:696`). +3. **Shrink full-extent scratch arrays to block extent**: `SZK_(GV)` → `max(1,nkblock)` for k-blocks + (`:2706`), or `SZI_/SZJ_(G)` → `niblock,njblock` for i/j tiles (`:621-636`). This is the memory + payoff: device/stack footprint is bounded by the *block* size, not the *domain* size. +4. **Introduce a block-local index** (`kk = k - ks + 1`, or `ii = i - i_start + 1 ; jj = j - j_start + 1`) + and rewrite every scratch-array reference from the global index to the local one, while grid + metric and full-size in/out arrays (`h_in`, `h_W`, `G%mask2dT`, ...) keep using the **global** + index unchanged (`:2754-2764`, `:717-718`). +5. **Pass the active sub-range `(ks,ke)` or `(i_start,i_end,j_start,j_end)` down** to any helper + subroutine that previously took the full range (`PPM_limit_pos(...,ks,ke)` vs the old + `PPM_limit_pos(...,nz)`; `zonal_flux_adjust(...,i_start,i_end,j_start,j_end,...)`). + +### 1.2 The i/j-tiled hybrid kernel (continuity `zonal_mass_flux`) + +`zonal_mass_flux`, `src/core/MOM_continuity_PPM.F90:696-736`, is the file's most fully evolved +example, combining i/j tiling with an explicit device-team launch: + +```fortran +do j_start=jsh,jeh,njblock ; do i_start=ish-1,ieh,niblock + i_end = min(i_start+niblock-1,ieh) + j_end = min(j_start+njblock-1,jeh) + + ! calculate number of teams + !$ nteams = ceiling(real((j_end-j_start+1)*(i_end-i_start+1))/128.) + + do concurrent (jj=1:j_end-j_start+1, ii=1:i_end-i_start+1) + do_I(ii,jj) = .true. + enddo + ! Set uh and duhdu. + !$omp target teams num_teams(nteams) + do k=1,nz + if (use_visc_rem) then + !$omp loop collapse(2) private(ii,jj) + do j=j_start,j_end ; do I=i_start,i_end + ii=I-i_start+1 ; jj=j-j_start+1 + visc_rem(ii,jj,k) = visc_rem_u(I,j,k) + enddo ; enddo + endif + !$omp loop collapse(2) private(ii,jj) + do j=j_start,j_end ; do i=i_start,i_end + ii=i-i_start+1 ; jj=j-j_start+1 + call flux_elem(u(i,j,k),h_in(i,j,k),h_in(i+1,j,k),h_W(i,j,k),h_W(i+1,j,k),h_E(i,j,k),& + h_E(i+1,j,k),uh_t(ii,jj,k),duhdu(ii,jj,k),visc_rem(ii,jj,k),G%dy_Cu(i,j),& + G%IareaT(i,j),G%IareaT(i+1,j),G%IdxT(i,j),G%IdxT(i+1,j),dt,CS%vol_CFL,& + por_face_areaU(I,j,k)) + ... + enddo ; enddo + enddo + !$omp end target teams + ... +enddo ; enddo +``` + +`flux_elem`/`flux_elem_OBC` are `elemental subroutine`s carrying `!DIR$ ATTRIBUTES FORCEINLINE` +(`MOM_continuity_PPM.F90:1086`, `:1149`); `ratio_max` is a plain `pure function` +(`MOM_continuity_PPM.F90:3086`) with **no** inline attribute of its own (nvfortran inlines it via the +`-Minline=name:ratio_max` build flag catalogued in `00-architecture.md` §7.5, not a source directive). +Either way they are called from inside the tile kernel — this is the "extract into `pure`/`elemental` +subroutine" idiom from `00-architecture.md` guiding principle 2, and it is what lets the tile body call +into shared code without a device-side v-table/module boundary. (Verified: the current merged form +uses Intel `!DIR$ ATTRIBUTES FORCEINLINE`, having replaced the earlier `!NVF$ INLINE` directive in +`93dbbd36e`; the hybrid kernel itself spans `:696-738`, ending at the `!$omp end target teams`.) + +### 1.3 Historical evolution of the same transformation (continuity) + +The current form is the end of a multi-commit evolution, all on `src/core/MOM_continuity_PPM.F90` +(`git log dev-gfdl..dev/gpu --oneline -- src/core/MOM_continuity_PPM.F90`, chronological): + +| Commit | What changed | +|---|---| +| `26b8b4da7` "implement tiling of zonal/meridional_mass_flux main loops" | **First** i/j tiling. Used **OpenACC** (`!$acc parallel loop`, `!$acc loop seq`) with hard-coded `TILE_SIZE_X=32,TILE_SIZE_Y=4` constants; introduced the `ii,jj` block-local index and block-shaped scratch arrays for the first time. | +| `3cb184edd` "use openmp instead of openacc" | Same tile structure, directives translated from `!$acc parallel/loop` to `!$omp target [teams] loop`. | +| `5b5f6b2b1` "add teams spec to problematic target region" | Added the hand-computed `nteams` and `num_teams(nteams)` — see §6. | +| `a327bbcf2` "continuity: runtime tile sizes user params" | `TILE_SIZE_X/Y` became runtime CS members read via `get_param`, defaulting to 0; `omp_get_num_devices()>0` checked at runtime to pick whole-domain sizing on GPU. | +| `bf3a6c3c3`, `e8b0ecfbf` "omp target teams loop -> do concurrent" | Reverted some `omp target teams loop` regions back to `do concurrent` where it performed better/was simpler. | +| `51c2f4032` "rename TILE_SIZE_[XY] to niblock/njblock" | Cosmetic rename to the current, more MOM6-idiomatic names. | +| `7f182139f` "proper ni/jblock selection in 3d_fluxes and adjust_vel" | Fixed missed call sites that weren't resolving 0→whole-domain. | +| `93dbbd36e` "Use blocking in k dimension for continuity reconstruction (#165)" | Added the **third, independent** blocking axis `nkblock` to the reconstruction routines (§1.1); switched the GPU/CPU default selection from a runtime `omp_get_num_devices()` check to a compile-time `#ifdef __NVCOMPILER_OPENMP_GPU` (§2). | + +Two lessons embedded in this history: (a) the port started on OpenACC and was mechanically +translated to OpenMP target once that became the house style — the tiling structure itself did not +change across that translation; (b) tile-size selection went through three different mechanisms +(hardcoded constant → runtime `omp_get_num_devices()` check → compile-time `#ifdef` + `if(block==0)` +resolution) before settling on the scheme described in §2. + +### 1.4 CoriolisAdv: the "serial-k-loop-with-2D-scratch" variant + +`MOM_CoriolisAdv.F90` (`b8c471cfa`, "Kblock coradcalc #167") starts from a different but very common +pre-existing pattern: a `do k=1,nz` **outer serial loop** (not a `do concurrent`) whose body computes +2-D (`SZIB_(G),SZJB_(G)`-shaped) work arrays fresh every iteration — the classic CPU-cache-friendly +"process one layer, reuse small 2-D scratch" idiom used throughout MOM6's dynamical core. Before +(`src/core/MOM_CoriolisAdv.F90`, pre-`b8c471cfa`): + +```fortran +real, dimension(SZIB_(G),SZJB_(G)) :: dvdx, dudy, ... ! 2-D, one layer's worth +... +do k=1,nz + do concurrent (J=Js_q:Je_q, I=Is_q:Ie_q) + dvdx(I,J) = (v(i+1,J,k)*G%dyCv(i+1,J)) - (v(i,J,k)*G%dyCv(i,J)) + dudy(I,J) = (u(I,j+1,k)*G%dxCu(I,j+1)) - (u(I,j,k)*G%dxCu(I,j)) + enddo + ... +enddo +``` + +After (`src/core/MOM_CoriolisAdv.F90:148-202` of the diff; current source `:373-`): + +```fortran +real, dimension(SZIB_(G),SZJB_(G),merge(GV%ke,CS%nkblock,CS%nkblock==0)) :: dvdx, dudy, ... +... +do k_start=1,nz,nkblock + k_end = min(k_start+nkblock-1, nz) + kmax = k_end - k_start + 1 + ... + do concurrent (kk=1:kmax, J=Js_q:Je_q, I=Is_q:Ie_q) DO_LOCALITY(local(k)) + k = k_start + kk - 1 + dvdx(I,J,kk) = (v(i+1,J,k)*G%dyCv(i+1,J)) - (v(i,J,k)*G%dyCv(i,J)) + dudy(I,J,kk) = (u(I,j+1,k)*G%dxCu(I,j+1)) - (u(I,j,k)*G%dxCu(I,j)) + enddo + ... +enddo +``` + +Same five-step recipe as §1.1, with one addition: `k` itself is no longer the `do concurrent` +control variable — the **block-local** index `kk` is (range `1:kmax`), and the *global* layer index +`k` is a per-iteration computed scalar carried via `DO_LOCALITY(local(k))`. This matters for +correctness (see §3) and was itself revised mid-flight: the local branch `kblock-coradcalc` +(single commit `0b5c00333`, predates the merged form) iterates the other way — `do concurrent +(k=kstart:kend, ...) ; kk = k - kstart + 1` — and a later commit inside `b8c471cfa` +("Iterate k-block loops on the block-local index directly") flipped it to iterate on `kk` with `k` +computed, which is the form now in `dev/gpu`. Both are bitwise-equivalent; the flip was a +style/performance preference, not a correctness fix (see §5 for why nvfortran can care about which +variable is the literal `do concurrent` control variable). + +Parts of `CorAdCalc` not yet portable (OBC segment handling, WENO, `KE_UP3`) are explicitly left as +serial `do k=k_start,k_end ! TODO: port (OBC GPU path not yet implemented)` loops with a manually +computed `kk`, i.e. the k-block **loop nest is always installed**, but its body may still be +call-by-call serial Fortran where a `do concurrent`/OpenMP rewrite hasn't happened yet. This is a key +structural point: k-blocking and "porting the body to a parallel construct" are separable steps. + +### 1.5 Horizontal viscosity: same template, still in-flight + +`kblock-hor-visc` (`36f51ff84` "block k in horizontal_viscosity") applies the *identical* recipe to +`MOM_hor_visc.F90`, and says so in its own commit message: *"Restructure the outer k-loop in +horizontal_viscosity into a ks-block loop using the same pattern as PPM_reconstruction_x/y."* +Representative before/after (`src/parameterizations/lateral/MOM_hor_visc.F90`, diff of `36f51ff84`): + +```fortran +! before +do k=1,nz + do concurrent (j=Jsq-1:Jeq+2, i=Isq-1:Ieq+2) + dudx(i,j) = CS%DY_dxT(i,j)*((G%IdyCu(I,j) * u(I,j,k)) - (G%IdyCu(I-1,j) * u(I-1,j,k))) + enddo + ... +enddo + +! after +do kstart=1,nz,nkblock + kend = min(kstart+nkblock-1, nz) + do concurrent (k=kstart:kend, j=Jsq-1:Jeq+2, i=Isq-1:Ieq+2) DO_LOCALITY(local(kk)) + kk = k - kstart + 1 + dudx(i,j,kk) = CS%DY_dxT(i,j)*((G%IdyCu(I,j) * u(I,j,k)) - (G%IdyCu(I-1,j) * u(I-1,j,k))) + enddo + ... +enddo +``` + +Note this file iterates on the *global* `k` (like `kblock-coradcalc`, not like the final merged +`CorAdCalc`), confirming both index styles are in active use and considered equivalent. Not-yet- +ported sub-blocks (e.g. `use_Leithy` smoothed-velocity terms, `id_normstress` diagnostic capture) are +left as literal serial `do k=kstart,kend ! TODO: port` loops nested inside the k-block loop — +exactly the same incremental-porting pattern seen in CoriolisAdv (§1.4). + +Follow-on commits on the same branch: +- `d6fe494e6` "pass nkblock as argument to horizontal_viscosity" — adds a `hor_visc_nkblock()` + accessor function so callers outside the module (which cannot see the `private` CS members + directly) can compute `merge(GV%ke, CS%nkblock, CS%nkblock==0)` and pass it in explicitly, rather + than the subroutine reaching into `CS%nkblock` itself. This is purely an encapsulation cleanup, not + a behavior change. +- `28eb296f4` "rearrange arrays to improve cpu perf" — see §4. (Not a *pure* declaration move — it + also fuses a short run of `do concurrent` loops and normalizes one array's dimension expression; + corrected there.) + +Commit order on the branch is confirmed `36f51ff84 → d6fe494e6 → 28eb296f4` (via `git merge-base +--is-ancestor`). **Caveat on staleness:** the `kblock-hor-visc` branch has since advanced well past +`28eb296f4` — its tip (`9b69fc581` at time of verification) carries ~15 further commits that extract +Leith/QG-Leith, EY24 backscatter, GME setup and the Leith+E update into named subroutines, add +tailored `DO_LOCALITY` clauses, and make `nkblock` a runtime parameter. Treat the three commits above +as the *illustrative* k-blocking slice of a still-evolving branch, not its final state. + +--- + +## 2. How `niblock`/`njblock`/`nkblock` get set + +All three block-size families follow one convention, first established for continuity and copied +verbatim to CoriolisAdv (`nkblock` only) and hor_visc (`nkblock` only): + +**(a) User-tunable runtime parameters**, declared as `integer` CS members and read with `get_param` +in the module's `*_init` routine: + +```fortran +! MOM_continuity_PPM.F90:3202-3213 +call get_param(param_file, mdl, "CONTINUITY_NIBLOCK", CS%niblock, ..., default=default_niblock, layoutParam=.true.) +call get_param(param_file, mdl, "CONTINUITY_NJBLOCK", CS%njblock, ..., default=default_njblock, layoutParam=.true.) +call get_param(param_file, mdl, "CONTINUITY_NKBLOCK", CS%nkblock, ..., default=default_nkblock, layoutParam=.true.) +if (CS%niblock < 0) call MOM_error(FATAL, "CONTINUITY_NIBLOCK must be nonnegative; use 0 to select the default block size.") +``` +(Analogously `CORIOLIS_ADV_NKBLOCK` in `MOM_CoriolisAdv.F90:2108-2112`, `HORVISC_NKBLOCK` in +`MOM_hor_visc.F90:2781-2785`.) Negative values are a fatal error; **0 is reserved to mean "dynamic / +whole extent."** + +**(b) Compile-time CPU-vs-GPU default divergence**, selected by the same preprocessor guard in every +module's `*_init` (`MOM_continuity_PPM.F90:3120-3129`): + +```fortran +#ifdef __NVCOMPILER_OPENMP_GPU + integer, parameter :: default_niblock = 0 !< whole domain / no cache blocking + integer, parameter :: default_njblock = 0 + integer, parameter :: default_nkblock = 0 +#else + ! These were found to give best performance in limited tests. + integer, parameter :: default_niblock = 32 + integer, parameter :: default_njblock = 4 + integer, parameter :: default_nkblock = 1 +#endif +``` +CoriolisAdv and hor_visc only have the k-axis, and their CPU default is `nkblock = 1` (i.e. exactly +the original per-layer serial loop, byte-for-byte the pre-port cache behavior) while GPU default is +`nkblock = 0`. + +**(c) Resolving `0` → whole extent, at call time, not at init time.** `CS%niblock`/`njblock`/`nkblock` +stay `0` in the CS; every call site that actually sizes an array or drives a loop resolves it locally: +- Continuity, i/j axis (`MOM_continuity_PPM.F90:186-189`, repeated at 4 call sites covering both + advection directions and both x-first/y-first branches): + ```fortran + LB = set_continuity_loop_bounds(G, CS, i_stencil=.false., j_stencil=.true.) + ! set whole-domain block sizes when ni/jblock is 0 + if (niblock == 0) niblock = LB%ieh-LB%ish+2 + if (njblock == 0) njblock = LB%jeh-LB%jsh+1 + ``` + The **resolved size depends on which of the four advection phases is being computed** (zonal vs. + meridional, symmetric-halo offset `+2` vs `+1`) — this is why the same `CS%niblock` is re-resolved + at every call site rather than once in `_init`. +- Continuity, k axis (`MOM_continuity_PPM.F90:512-513`, `557-558`, inside `zonal_edge_thickness` / + `meridional_edge_thickness`): `nkblock = CS%nkblock ; if (nkblock == 0) nkblock = nz`. +- CoriolisAdv (`MOM_CoriolisAdv.F90:275`): `nkblock = merge(GV%ke, CS%nkblock, CS%nkblock==0)` — a + one-line equivalent of the same `if`, used because it appears inside an array-dimension expression + (`real, dimension(SZIB_(G),SZJB_(G),merge(GV%ke,CS%nkblock,CS%nkblock==0))`, `:169`) as well as a + scalar assignment. +- hor_visc (`MOM_hor_visc.F90:500`, and — after `d6fe494e6` — via the external accessor + `hor_visc_nkblock(CS)` so callers outside the module can compute it too). + +This 3-stage design (user override → compile-time default → runtime 0-resolution) is what lets +*one source file* serve both roles: on CPU, tiles are small and cache-resident (`32×4` points, +1 layer); on GPU, "tiles" *are* the whole horizontal domain / whole column, so the tiling loops +degenerate to a single iteration and the inner kernel body becomes one big device-parallel region. + +**Evolutionary note:** this exact scheme is the fourth iteration of tile-size selection for +continuity (see §1.3): hardcoded constant → `omp_get_num_devices()` runtime check +(`a327bbcf2`) → `#ifdef __NVCOMPILER_OPENMP_GPU` compile-time default + runtime `if(block==0)` +resolution (`93dbbd36e`, the version now in the tree). The runtime `omp_get_num_devices()` check was +dropped because it doesn't compose with the k-axis default (nkblock had no such check at the time) +and because compile-time defaults are simpler to reason about when the same binary is not expected +to run on both host and device targets interchangeably. + +--- + +## 3. Why k/i/j-blocking preserves bitwise results + +The core argument, true in every one of the three cases studied: + +**Blocking only changes which *loop* an operation is nested inside and which *scratch buffer* holds +an intermediate value — it never changes what is computed from what.** For a fixed grid point +`(i,j,k)`: +- The set of *inputs* read (`h_in(i±1,j,k)`, `u(I,j,k)`, `G%mask2dT(i,j)`, ...) is identical before + and after blocking — blocking never changes a stencil's footprint. +- The *order of floating-point operations* within the expression computing that point's output + (`slp(...) = sign(1.,...) * min(abs(...), 2.*min(dMx,dMn))`, `h_W(i,j,k) = 0.5*(...) + + oneSixth*(...)`) is copied verbatim; only the array subscript used to store/load the intermediate + (`slp(i,j,k)` → `slp(i,j,kk)`) changes, and `kk` is a bijection of `k` within a block + (`kk = k - ks + 1`), so it addresses the *same logical value*, just at a different physical + offset. +- Each grid point's result therefore depends only on that point's own block-local computation, never + on which other points share its block or on block iteration order — blocks (and, within GPU + kernels, `do concurrent`/team iterations) can therefore be evaluated in *any* order, or all at + once, without changing any individual result. This is precisely why `do concurrent` (whose + standard-mandated semantics already forbid inter-iteration order dependence) is a legal target for + the transformation in the first place. +- Reductions are the one place order-independence needs an explicit argument, and the code handles it + two ways: (i) keep any cross-layer accumulation as a **sequential** `do k=1,nz` inside the tile, + so order is pinned by construction rather than left to a compiler reduction. Two distinct + accumulations in `zonal_mass_flux` do this: `visc_rem_max(ii,jj) = max(visc_rem_max, visc_rem(...,k))` + (`MOM_continuity_PPM.F90:746-752`) — a `max`, associative regardless of order — **and, more + importantly for the bitwise argument, a genuine floating-point vertical sum** + `uh_tot_0(ii,jj) = uh_tot_0 + uh_t(ii,jj,k)` / `duhdu_tot_0 += duhdu(ii,jj,k)` + (`MOM_continuity_PPM.F90:776-782`). The float sum is left as a literal serial `do k=1,nz` + precisely so its summation order is fixed; note the i/j tiling never splits this loop (each + `(ii,jj)` column accumulates its own independent partial sum over the *full* `1:nz`), so tiling + cannot reorder it — and continuity's k-axis blocking (`nkblock`) is confined to + `PPM_reconstruction_x/y`, which contains **no** cross-layer sum, so it cannot reorder it either. + (ii) where a + true reduction is used (`any_simple_OBC`, `MOM_continuity_PPM.F90:870`, + `DO_LOCALITY(reduce(.or.:any_simple_OBC))`), it is a boolean OR, which is associative/commutative + bit-for-bit regardless of grouping — unlike floating-point sums, boolean/integer reductions are + reorder-safe by construction. (Real floating-point reproducing sums are handled by an entirely + separate exact-integer mechanism in `MOM_coms.F90`; see `00-architecture.md` §7.2 and + `07-reproducibility.md` — none of the k/i/j-blocking transforms in this document touch a + floating-point reduction.) + +**Where care was genuinely needed** (documented in the diffs themselves): + +1. **Which variable is the `do concurrent` control variable vs. a private-computed scalar.** + CoriolisAdv's final form iterates on `kk` (block-local, `1:kmax`) and computes the global `k` + inside the loop body via `DO_LOCALITY(local(k))` — i.e. `k` must be declared with `local` locality + or every device thread would race on a shared `k`. Getting this wrong (declaring `k` shared instead + of `local`) would not change *which* value is stored where, but would be a data race / undefined + behavior on GPU, not a silent bitwise mismatch — still worth flagging because such bugs are easy to + introduce when converting a "compute index, then use it" idiom to a parallel loop. The + `DO_LOCALITY(local(...))` macro (`src/framework/do_concurrent_compat.h`) expands to the standard + `local()`/`reduce()` locality-specifier list when the compiler supports it + (`HAVE_FC_DO_CONCURRENT_LOCAL`), and to nothing otherwise — so on older compilers correctness + relies on the compiler's default (usually correct, but undocumented) treatment of loop-body scalars. +2. **Passing the active sub-range, not the full range, to helpers.** `PPM_limit_pos`/`PPM_limit_CW84` + changed signature from `(...,nz)` to `(...,ks,ke)` (`MOM_continuity_PPM.F90:2830` today vs. the + pre-`93dbbd36e` `(...,nz)`) specifically so a helper never iterates outside the block it was given + scratch data for — if it had kept iterating `1:nz` while `slp` was sized `nkblock`, it would read + garbage/out-of-bounds, not merely reorder arithmetic. This is a shape-safety concern introduced + *by* blocking, not a bitwise-order concern, but it is the actual bug class the PR's second commit + ("add error for -ve block size") and its general carefulness with `min(1,nkblock)`-sized arrays + guard against. +3. **Scratch array sizing must accommodate `nkblock==0` semantics.** `real, dimension(...,max(1,nkblock))` + (`MOM_continuity_PPM.F90:2706`) — the `max(1,...)` guards against a zero-sized array declaration + before `nkblock` has been resolved from `0` to `nz` inside the same subroutine (the resolution + happens a few lines earlier at `:512-513`, so in practice this is defensive, but it shows the + discipline expected: never let a `0` sentinel leak into an array bound unexamined). +4. **`-Minline` / `!NVF$ INLINE` on the per-point helpers.** `flux_elem`/`flux_elem_OBC`/`ratio_max` + are called from inside the blocked kernel and must be inlined onto the device. Commit + `93dbbd36e`'s "remove nvf inline and replace with intel forceinline" step is documented **only as a + performance change** — its verbatim message is *"Significantly improves performance of blocked + zonal/meridional_mass_flux at -O2"*, with no claim about correctness. (The earlier draft of this + doc attributed "Otherwise results are incorrect" to this commit; that phrase is **not** in the + commit message and has been removed.) The *correctness*-critical form of the inlining requirement + lives elsewhere: `00-architecture.md` §7.5 catalogues `-Minline=name:ratio_max,name:flux_elem` / + `!NVF$ INLINE` as "mandatory or wrong answers," and branch `kblock-hor-visc` even carries a + dedicated commit `4e3f1b758` "add !NVF$ INLINE to ratio_max flux_elem." So the load-bearing point + stands — the blessed pattern silently depends on force-inlining these helpers — but it should be + cited to the §7.5 workaround and `4e3f1b758`, not to `93dbbd36e`'s performance step. + > **Resolved (2026-07-14):** The primary source exists — it is `3cb184edd`'s own commit body: "for + > OpenMP, inlining of ratio_max and flux_elem is MANDATORY … Otherwise results are incorrect." (This + > doc's earlier pass searched `93dbbd36e` and found nothing, which is why the evidence looked + > missing.) It is era-specific evidence — the OpenACC→OpenMP translation, on the pre-`num_teams`-fix + > kernel — not a timeless language law, but treat it as binding for new code. + +--- + +## 4. Preserving CPU performance + +The mechanism is deliberately structural, not incidental: **when a block size resolves to a value +that reconstructs the pre-port loop nest exactly, CPU performance is preserved because the compiler +sees (almost) the same code.** + +- **`nkblock=1` on CPU (CoriolisAdv, hor_visc)** literally reconstructs `do k_start=1,nz,1 ; k_end = + k_start` — a trivial one-layer-at-a-time outer loop identical in trip count and body to the + pre-blocking `do k=1,nz`, with 2-D-sized scratch arrays (`nkblock=1` makes the 3rd dimension size 1, + which the compiler can treat as effectively 2-D). No cache-blocking gain is *lost* because the + original code was already "blocked" at the finest possible granularity (one layer) by construction. +- **`niblock=32, njblock=4, nkblock=1` on CPU (continuity)** — the code comment is explicit: *"These + were found to give best performance in limited tests"* (`MOM_continuity_PPM.F90:3125`). This is a + genuine, non-default cache-blocking regime (32×4-point horizontal tiles processed one at a time) + chosen empirically, not a degenerate case of "whole domain" — i.e. for continuity, CPU performance + is preserved by *actually* cache-blocking, not merely by not-blocking. +- **Commit `26b8b4da7`'s own log entry** ("seems to be roughly same gpu perf, but much better CPU + perf") is the origin evidence that this i/j-tiling axis exists *specifically* for CPU cache + behavior — GPU performance was roughly a wash, but CPU improved markedly, which is exactly the + trade the "preserve both" strategy is built to capture. + +**The `28eb296f4` "rearrange arrays to improve cpu perf" commit** (on `kblock-hor-visc`, 72 insertions +/ 80 deletions in `MOM_hor_visc.F90`) is **overwhelmingly, but not purely,** a declaration-order +change. I re-derived its full diff (verification); it does three things: +1. **Declaration reordering (the bulk).** Large `real, dimension(...,nkblock)` blocks + (`Del2u/h_u/...`, `dvdx/dudy/...`, `div_xx/sh_xx/str_xx/...`, `Ah/Kh/Shear_mag/...`) are lifted out + of their old positions and re-emitted, verbatim, lower in the local-variable block (adjacent to the + `Ah_q/Ah_h` full-`SZK` arrays). No statement that computes a value is touched by this part. +2. **One dimension-expression normalization.** `str_xy_BS` changes from + `dimension(SZIB_(G),SZJB_(G),merge(GV%ke,CS%nkblock,CS%nkblock==0))` to + `dimension(SZIB_(G),SZJB_(G),nkblock)`. This is **semantically identical** *because* + `d6fe494e6` already made `nkblock` a dummy argument the caller sets to exactly that `merge(...)` + value — so the two expressions have the same runtime extent. It is a cleanup, not a shape change. +3. **A genuine loop fusion (the one real structural change).** Three consecutive + `do concurrent (k=kstart:kend, j=Jsq-1:Jeq+2, i=Isq-1:Ieq+2)` loops — computing `dudx(i,j,kk)`, + then `dvdy(i,j,kk)`, then `sh_xx(i,j,kk) = dudx(i,j,kk) - dvdy(i,j,kk)` in three separate passes — + are **merged into a single `do concurrent`** with all three assignments in one body (the + `@@ -724,16 +724,8 @@` hunk, −8 net lines). So the earlier characterization of "zero algorithmic + diff / no change to any statement" was **incorrect**: the loop *nesting* changed. + +**This fusion is nonetheless bitwise-safe**, and it is worth stating why explicitly (this is exactly +the "fused vs. split loops" hazard flagged during verification): the three fused loops share the +*identical* iteration space `(k=kstart:kend, j=Jsq-1:Jeq+2, i=Isq-1:Ieq+2)`; each writes a *distinct* +array (`dudx`, `dvdy`, `sh_xx`); and the only intra-set dependence — `sh_xx(i,j,kk)` reading +`dudx(i,j,kk)` and `dvdy(i,j,kk)` — is at the **same** `(i,j,kk)` computed earlier in the same fused +iteration, never at a neighbor or a different layer. Fusing therefore neither reorders any +floating-point operation within an expression nor introduces a cross-iteration read-after-write, so +every point's result is bit-identical. (A fusion that instead pulled a *reduction* or a +*neighbor-stencil* read across the merged boundary would **not** be safe — that is the case to watch +for when imitating this commit.) + +The declaration-order half of the commit is real and load-bearing on its own. The pre-existing source +warns about it (comment quoted verbatim below, confirmed present in the tree): + +```fortran +! NOTE: The position of these declarations can impact performance, due to the +! very large number of stack arrays in this function. Move with caution! +``` + +This confirms that for a routine with dozens of large automatic (stack) arrays now carrying an extra +`nkblock` dimension, **the compiler's stack layout/alignment decisions are sensitive to declaration +order**, and this sensitivity is large enough to be worth a dedicated commit — i.e. k-blocking's +memory-footprint increase (2-D scratch → 3-D scratch-of-size-`nkblock`) can itself regress CPU +performance through stack-layout effects unrelated to the algorithm, and the fix is non-obvious +(reordering declarations, not touching logic). This is the single most surprising CPU-perf lesson in +the three case studies: **not every regression from k-blocking is about cache blocking per se — some +are about how many/how-large stack arrays a single Fortran procedure declares and in what order.** + +--- + +## 5. Contrasting the three cases + +| | Continuity (`MOM_continuity_PPM.F90`) | CoriolisAdv (`MOM_CoriolisAdv.F90`) | hor_visc (`MOM_hor_visc.F90`) | +|---|---|---|---| +| Status | Merged, most mature | Merged (`b8c471cfa`) | In-flight (`kblock-hor-visc`) | +| Blocking axes | i, j, **and** k (three independent CS members) | k only | k only | +| Pre-blocking loop shape | `do concurrent(k,j,i)` over the whole 2-D reconstruction plane per call, or a hand-tiled `!$omp target teams` region | serial `do k=1,nz` outer loop, 2-D scratch reused each iteration | serial `do k=1,nz` outer loop, 2-D scratch reused each iteration (identical shape to CoriolisAdv pre-port) | +| Block-local index style | `ii,jj` for i/j tiles; `kk` for k-blocks, computed as `kk = k - ks + 1` inside a `do concurrent(k=ks:ke,...)` (k is the control variable) | Iterates on `kk` (`1:kmax`) as the control variable, computing `k = k_start + kk - 1` via `DO_LOCALITY(local(k))` — the *opposite* convention from continuity | Iterates on the *global* `k` as control variable (same convention as continuity), like the pre-merge `kblock-coradcalc` branch | +| Device directive at the outer/whole-tile level | Explicit `!$omp target teams num_teams(nteams)` with hand-computed team count (§6) | Plain `do concurrent`, no explicit team/thread directives seen at this level | Plain `do concurrent`, matching CoriolisAdv | +| Incremental-porting markers | None needed — reconstruction is fully ported | `! TODO: port (OBC GPU path not yet implemented)` serial fallback loops nested inside the k-block loop | `! TODO: port` serial fallback loops (`use_Leithy`, `id_normstress`) nested inside the k-block loop, same idiom | +| CPU-perf-specific follow-up commit | None beyond the empirically tuned `32/4/1` defaults | None found | `28eb296f4` "rearrange arrays to improve cpu perf" — mostly declaration reordering + one bitwise-safe loop fusion (§4) | +| Param names | `CONTINUITY_NIBLOCK`/`NJBLOCK`/`NKBLOCK` | `CORIOLIS_ADV_NKBLOCK` | `HORVISC_NKBLOCK` | + +**What is common (the blessed template, restated):** one (or three) CS-level integer block-size +parameter(s), defaulting to a small CPU-tuned constant (or `1` for pure k-blocking) on CPU builds and +`0` ("whole extent") on `__NVCOMPILER_OPENMP_GPU` builds; an outer host loop striding over blocks; +block-sized (not domain-sized) scratch arrays; a block-local index computed as an offset from the +block start; the active sub-range threaded explicitly into any helper subroutine; `do concurrent` +(with `DO_LOCALITY` macros for locality clauses) as the default parallel construct for the block +body. + +**What differs:** (a) continuity alone combines all three axes and is the only one with an explicit +manual-team-count `omp target teams` region — the other two modules haven't (yet, as of this +writing) needed to hand-tune team counts; (b) the two k-block-only modules disagree with each other +on which of `k`/`kk` is the literal `do concurrent` control variable, showing this is a +non-load-bearing style choice, not part of the "blessed" contract; (c) only hor_visc has hit a +declaration-order CPU-perf regression, plausibly because it is the largest/most stack-array-heavy of +the three routines. + +--- + +## 6. `!$omp target teams num_teams(nteams)` vs. plain `do concurrent` + +The default parallel idiom across the whole port is `do concurrent` (`00-architecture.md` guiding +principle 3). The blocked kernels in continuity's `zonal_mass_flux`/`meridional_mass_flux` are the +one place in these three case studies where an explicit `!$omp target teams num_teams(nteams)` region +wraps a serial-looking `do k=1,nz` with `!$omp loop collapse(2)` inside — and the reason is recorded +verbatim in the commit that introduced it, `5b5f6b2b1` ("add teams spec to problematic target +region"): + +> For some reason omp runtime was only starting a kernel with 17 blocks when the openacc version +> would start it with 238 or something like that. Manually calculating number of teams sped it up. + +I.e. this is a **documented nvfortran OpenMP-runtime under-launch bug**, not a general preference for +`target teams` over `do concurrent`. The workaround: + +```fortran +! calculate number of teams +!$ nteams = ceiling(real((j_end-j_start+1)*(i_end-i_start+1))/128.) +... +!$omp target teams num_teams(nteams) +do k=1,nz + !$omp loop collapse(2) private(ii,jj) + do j=j_start,j_end ; do i=i_start,i_end + ... +``` +`128` is chosen as a thread-block-size divisor (an early revision of this same commit briefly also +carried `thread_limit(128)`, later dropped — current source at `:707` uses `num_teams(nteams)` alone +and relies on the runtime's default thread count per team). The `!$` sentinel is a conditional +OpenMP-compilation comment, so `nteams` is only computed/used when actually building with OpenMP. + +**Decision rule observed in the source, in priority order:** +1. **Default to `do concurrent`** for any loop whose iteration space can be expressed that way and + where the compiler's own scheduling has not been shown to misbehave (this is the overwhelming + majority of loops in all three modules, including most of the blocked bodies themselves, e.g. the + `do concurrent(jj=1:...,ii=1:...)` initializations flanking the teams region above). +2. **Escalate to `!$omp target [teams] loop`** when a region needs a reduction `do concurrent` can't + (yet, portably) express, or when `do concurrent` alone under-parallelizes/misschedules on + nvfortran (the general pattern named in `00-architecture.md` guiding principle 3 and its §7.5 + compiler-workaround catalogue, e.g. `e8b0ecfbf` "omp target teams loop -> do concurrent" shows the + traffic runs both directions depending on measured behavior). +3. **Escalate further to an explicit, hand-computed `num_teams(nteams)`** only when even + `target teams loop`'s automatic team count is empirically wrong (measured: 17 launched vs. ~238 + expected) — i.e. this is a last-resort, bug-specific fix applied to exactly the one region in + continuity where it was needed, not a house style to imitate by default. Neither CoriolisAdv nor + hor_visc's k-blocking (as of the branches studied here) needed this escalation, consistent with + §5's observation that the manual team count is a continuity-specific quirk, not part of the + general k-blocking template. + +--- + +## 7. The blessed k-blocking recipe — a followable template + +This is the prescriptive form of §1–§6: a numbered procedure for k-blocking one serial-`k` loop nest, +with the exact code shapes to copy. Every step cites a merged (or near-merged) example. Substitute +your module's name for `MYMOD` and choose a `MYMOD_NKBLOCK` param name. (For i/j tiling, the same +skeleton applies with `niblock`/`njblock`; only continuity currently needs it — see §1.2.) + +### 7.0 Pre-flight checklist — does this loop nest qualify? + +A loop nest is a k-blocking candidate **iff all** of these hold: + +- [ ] It is (or can be) an **outer `do k=1,nz`** whose body is a horizontal (2-D, `i`/`j`) calculation + — the classic "process one layer, reuse small 2-D scratch" idiom (`MOM_CoriolisAdv.F90` and + `MOM_hor_visc.F90` pre-port). A nest that is already a single `do concurrent (k,j,i)` over the + whole cube (continuity reconstruction) also qualifies — you are bounding its scratch, not + serializing it. +- [ ] The per-point computation is **pointwise in `k`**: point `(i,j,k)`'s output depends only on + inputs at layer `k` (any horizontal stencil is fine). **No cross-layer coupling** — no + `k`↔`k±1` dependence, no running vertical integral, no tridiagonal-in-`k` solve. (Those belong to + the `vert_friction` "teams-loop with serial inner k" pattern, `00-architecture.md` §4.3, **not** + here.) +- [ ] Any **vertical reduction** in the body is either (a) associative-by-construction (`max`, `.or.`, + integer) or (b) a float sum you can keep as a **sequential `do k=1,nz`** that blocking will + *not* split (see §3(i)). If a float sum would have to be partitioned across blocks, **stop** — + that reorders arithmetic and breaks bitwise reproducibility. + +**Disqualifiers** (do *not* k-block; port differently or leave serial): implicit vertical solves +(tridiagonal), cumulative `k` integrals, remapping/regridding across layers, or anything where a +block boundary would fall *inside* a summation or a `k`-stencil. + +### 7.1 Step-by-step + +**Step 1 — Add the CS member.** In `MYMOD_CS`: +```fortran +integer :: nkblock !< The k block size used in <...> calculations [nondim]. +``` +(`MOM_continuity_PPM.F90:78`, `MOM_CoriolisAdv.F90:59`, `MOM_hor_visc.F90:125`.) + +**Step 2 — Compile-time CPU/GPU default divergence**, in `MYMOD_init`, before the `get_param`: +```fortran +#ifdef __NVCOMPILER_OPENMP_GPU + integer, parameter :: default_nkblock = 0 !< whole column / no cache blocking +#else + integer, parameter :: default_nkblock = 1 !< one layer at a time = pre-port CPU behavior +#endif +``` +(`MOM_continuity_PPM.F90:3120-3129`, `MOM_CoriolisAdv.F90:2094-2098`, `MOM_hor_visc.F90:2759-2763`.) +Continuity's i/j axis uses `32`/`4` instead of `1`; the k axis is always `0` (GPU) / `1` (CPU). + +**Step 3 — `get_param`, with the `0`-is-fatal-if-negative guard:** +```fortran +call get_param(param_file, mdl, "MYMOD_NKBLOCK", CS%nkblock, & + "The k-direction block size ... the default 0 setting is dynamic and fits the "//& + "full vertical column.", default=default_nkblock, layoutParam=.true.) +if (CS%nkblock < 0) call MOM_error(FATAL, "MYMOD_NKBLOCK must be >= 0.") +``` +(`MOM_continuity_PPM.F90:3210-3220`, `MOM_CoriolisAdv.F90:2108-2112`, `MOM_hor_visc.F90:2781-2785`.) +`0` is reserved to mean "dynamic / whole extent"; negative is a hard error. + +**Step 4 — Resolve `0` → whole extent at the point of use** (never store the resolved value back into +`CS`). Two equivalent forms: +```fortran +nkblock = CS%nkblock ; if (nkblock == 0) nkblock = nz ! scalar form (continuity :512-513) +nkblock = merge(GV%ke, CS%nkblock, CS%nkblock==0) ! expression form (CoriolisAdv :275) +``` +Use the `merge(...)` form when the value must also appear in an **array-dimension expression** (Step 5). +If the loop nest lives in a *different* module from the CS, expose an accessor rather than reaching +into private members — `hor_visc_nkblock(CS)` returns `merge(GV%ke,CS%nkblock,CS%nkblock==0)` for +external callers (`d6fe494e6`, `MOM_hor_visc.F90:274,298`). + +**Step 5 — Shrink full-column scratch to one block.** Every `SZK_(GV)`/`nz`-deep automatic work array +gets its 3rd dimension replaced by the block size, guarded against a zero-size declaration: +```fortran +real, dimension(SZIB_(G),SZJB_(G),merge(GV%ke,CS%nkblock,CS%nkblock==0)) :: dvdx, dudy, ... ! CoriolisAdv :169 +! or, when nkblock is already a resolved local/argument: +real, dimension(SZI_(G),SZJ_(G),max(1,nkblock)) :: slp ! continuity :2706 +``` +The `merge(...)`/`max(1,...)` guard exists so a `0` sentinel can never reach an array bound (§3(3)). +**Stack-layout caution:** in a procedure with many such arrays, *where* you declare them can itself +move CPU performance — keep related blocks together and expect to tune order empirically (§4, +`28eb296f4`; the source comment "Move with caution!"). + +**Step 6 — Wrap the `k` axis in an outer host loop over block starts:** +```fortran +do k_start=1,nz,nkblock + k_end = min(k_start+nkblock-1, nz) + kmax = k_end - k_start + 1 + ... +enddo +``` +(`MOM_CoriolisAdv.F90:373-374`, `MOM_continuity_PPM.F90` reconstruction uses `ks/ke`, `MOM_hor_visc.F90` +uses `kstart/kend`.) i/j tiling nests two such loops (`do j_start=...; do i_start=...`, +`MOM_continuity_PPM.F90:696`). + +**Step 7 — The device kernel body + block-local index.** Convert the body to `do concurrent` with a +block-local index; **either** convention is accepted (they are bitwise-equivalent, §1.4/§5): +```fortran +! (a) iterate the block-local index (merged CoriolisAdv b8c471cfa :382): +do concurrent (kk=1:kmax, J=Js_q:Je_q, I=Is_q:Ie_q) DO_LOCALITY(local(k)) + k = k_start + kk - 1 + dvdx(I,J,kk) = (v(i+1,J,k)*G%dyCv(i+1,J)) - (v(i,J,k)*G%dyCv(i,J)) +enddo +! (b) iterate the global k (continuity :2753, hor_visc, kblock-coradcalc): +do concurrent (k=ks:ke, j=jsl:jel, i=isl:iel) DO_LOCALITY(local(h_im1,h_ip1,kk)) + kk = k - ks + 1 + h_W(i,j,kk-or-k) = ... +enddo +``` +**Index-mapping rule (do not get this wrong):** *scratch/work* arrays use the **block-local** third +index (`kk`); *full-size in/out and grid-metric* arrays (`u`, `v`, `h_in`, `h_W`, `G%dyCv`, +`G%mask2dT`) keep the **global** `k`. Whichever variable is *not* the `do concurrent` control variable +must be declared with `DO_LOCALITY(local(...))` so each device iteration owns a private copy (a shared +scalar here is a data race on GPU — §3(1)). The `DO_LOCALITY` macro +(`src/framework/do_concurrent_compat.h`) degrades to nothing where `HAVE_FC_DO_CONCURRENT_LOCAL` is +unset. + +**Step 8 — Thread the active sub-range into every helper** the body calls. Change signatures that took +the full range to take `(k_start,k_end)` (or `ks,ke` / `i_start,i_end,j_start,j_end`): +```fortran +call PPM_limit_pos(h_in, h_W, h_E, h_min, G, GV, isl, iel, jsl, jel, ks, ke) ! was (...,nz) +call gradKE(u, v, h, KE, KEx, KEy, k_start, k_end, nkblock, G, GV, US, CS) ! CoriolisAdv :781 +``` +A helper that kept iterating `1:nz` against block-sized scratch would read out of bounds — this is a +shape-safety bug, not merely a reordering (§3(2)). + +**Step 9 — Incremental porting is allowed.** If a sub-block of the body isn't ready for `do +concurrent` (OBC paths, WENO, `use_Leithy`, diagnostic capture), leave it as a **serial** loop *inside* +the k-block loop with a hand-computed `kk`, tagged for follow-up: +```fortran +do k=k_start,k_end ! TODO: port (OBC GPU path not yet implemented) + kk = k - k_start + 1 + ... +enddo +``` +The k-block *nest* is installed unconditionally; porting each body to a parallel construct is a +separable later step (§1.4, §1.5). + +**Step 10 — Device data mapping** for any new CS arrays follows the standard pattern (unchanged by +blocking): `ALLOC_(CS%x(...)); CS%x=0.0; !$omp target enter data map(to: CS%x)` in `_init`, mirrored +`!$omp target exit data map(delete: CS%x)` beside `DEALLOC_` in `_end` (`00-architecture.md` §9). +Block-sized *automatic* scratch (Step 5) needs no explicit map when used only inside `do concurrent`; +an explicit hand-launched `!$omp target teams` region maps its scratch with `enter/exit data +map(alloc:/release:)` around the tile loop (continuity `slp`, `MOM_continuity_PPM.F90:2708` / +`:2835`). + +**Step 11 (escalation, rarely needed) — hand-tuned team count.** Default to `do concurrent`. Only if +profiling shows nvfortran under-launching teams for a hand-written `!$omp target teams` region do you +add a manual `num_teams` (continuity's `zonal_mass_flux` is the *sole* case in these three modules, +§6): `!$ nteams = ceiling(real((j_end-j_start+1)*(i_end-i_start+1))/128.)` then +`!$omp target teams num_teams(nteams)`. This is a bug-specific last resort (`5b5f6b2b1`), not a +default to imitate. + +### 7.2 Verification step (mandatory before claiming a port) + +k-blocking is a bitwise-preserving refactor, so it is verifiable *exactly* — there is no "close +enough": + +1. Build the **pre-change** binary and the **post-change** binary at the same optimization level. +2. Run both (CPU is sufficient for the arithmetic check; `nkblock=1` on CPU reconstructs the pre-port + loop, §4) and compare `MOM_checksums` field checksums (`hchksum`/`uchksum`/`vchksum`, + `popcnt`-based, `MOM_checksums.F90`, `00-architecture.md` §7.2) at matching timesteps. **Every + checksum must be bit-identical**; a single differing bit means the transform reordered arithmetic + (most likely a split float sum, a mishandled reduction, or a helper still iterating the full range). +3. Compare the EFP reproducing-sum energy output (`write_energy`) — also required to be bit-identical. +4. Confirm the `nkblock=0` (GPU-default) and `nkblock=nz` (explicit) resolutions agree with each other + and with `nkblock=1`, since all three must produce identical results by construction. +5. Sanity-check the `!NVF$ INLINE`/`-Minline` requirement for any per-point helper called from the + kernel — dropping it can change answers on nvfortran (§3(4), `00-architecture.md` §7.5). + +If any checksum diverges, the port is wrong — do not "accept" a nonzero diff as rounding. + +--- + +## 8. Where to look next + +- `src/core/MOM_continuity_PPM.F90:76-78,164-421,502-736,2684-2989,3110-3223` — CS block-size + members, `niblock`/`njblock` resolution at all four `continuity_PPM`/`continuity_3d_fluxes`/ + `continuity_adjust_vel` call sites, the hybrid tiled kernel, `PPM_reconstruction_x/y`, and + `continuity_PPM_init` defaults. +- `src/core/MOM_CoriolisAdv.F90:59,169-275,373-` — `nkblock` CS member, `merge(GV%ke,CS%nkblock,...)` + resolution pattern, the k-block main loop. +- `src/parameterizations/lateral/MOM_hor_visc.F90` on branch `kblock-hor-visc` (commits `36f51ff84`, + `d6fe494e6`, `28eb296f4`) — same template applied to the largest of the three routines, plus the + declaration-order CPU-perf lesson. +- `src/framework/do_concurrent_compat.h` — the `DO_LOCALITY(X)` macro used throughout to make + `local()`/`reduce()` locality clauses conditional on `HAVE_FC_DO_CONCURRENT_LOCAL`. +- Git: `git log dev-gfdl..dev/gpu --oneline -- src/core/MOM_continuity_PPM.F90` for the full + continuity evolution; `git show b8c471cfa`; `git diff dev/gpu...kblock-coradcalc`; `git log + dev/gpu..kblock-hor-visc --oneline` and `git diff dev/gpu...kblock-hor-visc -- src/parameterizations/lateral/MOM_hor_visc.F90`. + +--- + +## Verification notes + +Opus verification pass against source + git (no build/run). Baseline `dev-gfdl`, branch `dev/gpu`. + +### Confirmed (checked directly in code/git) + +- **Continuity CS members** `niblock`/`njblock`/`nkblock` at `MOM_continuity_PPM.F90:76-78`. ✓ +- **`#ifdef __NVCOMPILER_OPENMP_GPU` default divergence** `0/0/0` (GPU) vs `32/4/1` (CPU) at + `:3120-3129`; `get_param` for `CONTINUITY_NIBLOCK/NJBLOCK/NKBLOCK` with negative-is-FATAL at + `:3202-3223`. ✓ +- **`if (niblock==0) niblock = LB%ieh-LB%ish+2` whole-domain fill** at `:186-189`, and the claim that + the resolved size is re-derived per advection phase (`+2` vs `+1`) — confirmed, four phases. ✓ +- **Hybrid `num_teams` kernel** at `:696-738` (doc previously said `:696-736`; corrected to `:738`, + the `!$omp end target teams`), with `!$ nteams = ceiling(real((...)*(...))/128.)`. ✓ +- **PPM_reconstruction blocking** (`93dbbd36e`): `nkblock` added as subroutine argument + (`:2684`), `slp` shrunk to `dimension(...,max(1,nkblock))` (`:2706`), outer `do ks=1,nz,nkblock`, + `kk=k-ks+1`, and `PPM_limit_pos/CW84` signature changed `(...,nz)` → `(...,ks,ke)` — all verified + against the pre-`93dbbd36e` source (which had `slp(SZI,SZJ,SZK)` and `PPM_limit_pos(...,nz)`). ✓ +- **k-axis `0`→`nz` resolution** at `:512-513`, `:557-558`; the "one k-BLOCK of scratch" bitwise + argument (pointwise in `k`, no cross-layer sum in reconstruction) — re-derived and holds. ✓ +- **CoriolisAdv `b8c471cfa`** iterates `kk=1:kmax`, `k=k_start+kk-1`, `DO_LOCALITY(local(k))` + (`:373-410`); `merge(GV%ke,CS%nkblock,CS%nkblock==0)` in both array dims (`:169,181,195,203`) and + scalar (`:275`); `CORIOLIS_ADV_NKBLOCK` at `:2108-2112`. ✓ +- **kk-vs-k divergence** vs branch `kblock-coradcalc` (`0b5c00333`): confirmed the branch iterates + `do concurrent (k=kstart:kend,...) ; kk=k-kstart+1` (opposite convention), and a commit inside + `b8c471cfa` flipped it to iterate `kk`. Both bitwise-equivalent. ✓ +- **hor_visc branch** commit order `36f51ff84 → d6fe494e6 → 28eb296f4` (via `merge-base + --is-ancestor`); `36f51ff84` blocks k iterating the *global* `k`; `d6fe494e6` adds the + `hor_visc_nkblock` accessor and makes `nkblock` a dummy arg; `HORVISC_NKBLOCK` at `:2781-2785`. ✓ +- **"Move with caution!" source comment** — exact text confirmed present in the branch tree. ✓ +- **`5b5f6b2b1` num_teams under-launch** rationale — commit message quote confirmed. ✓ + +### Corrected + +1. **`28eb296f4` is NOT "zero algorithmic diff" (critical).** Re-deriving its full 72/80-line diff + shows, besides the declaration reordering: (a) `str_xy_BS`'s dimension normalized + `merge(GV%ke,CS%nkblock,...)` → `nkblock` (equivalent post-`d6fe494e6`), and (b) a **genuine loop + fusion** — three separate `do concurrent` loops (`dudx`, `dvdy`, `sh_xx`) merged into one + (`@@ -724,16 +724,8 @@`). Rewrote §4 to state this and to prove the fusion is nonetheless + bitwise-safe (identical iteration space, distinct output arrays, only a same-index intra-iteration + dependence). This is exactly the "fused vs split loops" hazard the task asked to hunt for; the + fusion is safe *here*, but the doc now spells out what would make it unsafe. +2. **§3(4) misquoted `93dbbd36e`.** The phrase "Otherwise results are incorrect" is **not** in the + commit message (which says only "Significantly improves performance ... at -O2"). Reattributed the + correctness-critical inlining claim to `00-architecture.md` §7.5 and branch commit `4e3f1b758`. + Its primary source has since been located in `3cb184edd`'s commit body (see §3(4)). +3. **§1.2 helper attributes.** `ratio_max` is a `pure function` (`:3086`) with **no** FORCEINLINE + directive; only `flux_elem`/`flux_elem_OBC` are `elemental subroutine`s carrying + `!DIR$ ATTRIBUTES FORCEINLINE` (`:1086`, `:1149`). Corrected the "both elemental + FORCEINLINE" + generalization. +4. **§3(i) mislabel.** The `:775-782` accumulation is not `visc_rem_max` — it is a genuine + floating-point vertical **sum** `uh_tot_0/duhdu_tot_0 += ...(k)`. Corrected and used it as the + stronger bitwise example (a real FP sum kept as a sequential `do k=1,nz` that tiling never splits). + +### Enhancements + +- Added §7 "The blessed k-blocking recipe" — an 11-step followable template (CS member → ifdef default + → get_param → 0-resolution → scratch reshaping → outer loop → device kernel + index-mapping rule → + helper sub-range → incremental-porting fallback → data mapping → team-count escalation), each step + cited to a merged example; plus a §7.0 pre-flight qualify/disqualify checklist and a §7.2 checksum + verification procedure. +- Added a branch-staleness caveat (kblock-hor-visc tip `9b69fc581` is ~15 commits past `28eb296f4`). + +### Confidence + +**High** on all continuity and CoriolisAdv claims (verified verbatim against current tree and +pre-commit source). **High** on the `28eb296f4` correction (full diff inspected; loop fusion and +dimension-normalization are unambiguous in the hunk). **High** on bitwise-preservation for all three +cases, including the fused loop. The provenance of the "inline-or-wrong-answers" claim is now settled +(`3cb184edd`'s commit body, §3(4)), confirming the structural conclusion that force-inlining is +required. diff --git a/knowledge/gpu-knowledge/06-eos-layer.md b/knowledge/gpu-knowledge/06-eos-layer.md new file mode 100644 index 0000000..e51ff71 --- /dev/null +++ b/knowledge/gpu-knowledge/06-eos-layer.md @@ -0,0 +1,647 @@ +# The EOS layer: runtime polymorphism vs. nvfortran (dev/gpu) + +> Drills into architecture doc §7.1. Scope: `src/equation_of_state/`. This is the canonical +> case study for "`class(*)`/runtime polymorphism is a disaster on device" — read this before +> touching any polymorphic dispatch elsewhere in the tree. + +--- + +## 1. The old dispatch: `EOS_type` → `class(EOS_base)` → deferred elemental procedures + +### 1.1 The wrapper type + +`EOS_type` (`MOM_EOS.F90:117-164`) is a plain (non-polymorphic) derived type holding scalar +parameters (unit-scaling factors, freezing-point coefficients) plus one polymorphic component: + +```fortran +!> A control structure for the equation of state +type, public :: EOS_type ; private + integer :: form_of_EOS = 0 + ... + !> The instance of the actual equation of state + class(EOS_base), allocatable :: type +end type EOS_type +``` +(`MOM_EOS.F90:117`, component at `:162`) + +### 1.2 The abstract base with deferred elemental bindings + +`EOS_base` (`MOM_EOS_base_type.F90:13-74`) is `abstract` and declares nine `deferred` type-bound +procedures, all `elemental`, all taking the polymorphic `this` as their first dummy: + +```fortran +type, abstract :: EOS_base +contains + procedure(i_density_elem), deferred :: density_elem + procedure(i_density_anomaly_elem), deferred :: density_anomaly_elem + procedure(i_spec_vol_elem), deferred :: spec_vol_elem + procedure(i_spec_vol_anomaly_elem), deferred :: spec_vol_anomaly_elem + procedure(i_calculate_density_derivs_elem), deferred :: calculate_density_derivs_elem + procedure(i_calculate_density_second_derivs_elem), deferred :: calculate_density_second_derivs_elem + procedure(i_calculate_specvol_derivs_elem), deferred :: calculate_specvol_derivs_elem + procedure(i_calculate_compress_elem), deferred :: calculate_compress_elem + procedure(i_EOS_fit_range), deferred :: EOS_fit_range + ! shared, non-deferred fallbacks provided by the base module: + procedure :: calculate_density_array_2d => a_calculate_density_array_2d + procedure :: calculate_density_array_3d => a_calculate_density_array_3d + procedure :: calculate_density_derivs_2d => a_calculate_density_derivs_2d + procedure :: calculate_density_derivs_3d => a_calculate_density_derivs_3d + procedure :: calculate_density_second_derivs_2d => a_calculate_density_second_derivs_2d + ... +end type EOS_base +``` +(`MOM_EOS_base_type.F90:13-74`) + +Every deferred interface is `elemental` and carries `class(EOS_base), intent(in) :: this` as the +first argument (e.g. `i_density_elem`, `:81-88`). The **base fallbacks** (`a_calculate_density_array_2d` +at `:268`, `a_calculate_density_array_3d` at `:298`, `a_calculate_density_derivs_2d/3d` at `:428/456`, +`a_calculate_density_second_derivs_2d` at `:542`) all have the same shape: take a whole array section, +and apply the elemental deferred procedure to it *through `this`*: + +```fortran +! a_calculate_density_array_3d, MOM_EOS_base_type.F90:298-326 +subroutine a_calculate_density_array_3d(this, T, S, pressure, rho, dom, rho_ref) + class(EOS_base), intent(in) :: this + ... + if (present(rho_ref)) then + rho(is:ie, js:je, ks:ke) = this%density_anomaly_elem(T(is:ie, js:je, ks:ke), & + S(is:ie, js:je, ks:ke), pressure(is:ie, js:je, ks:ke), rho_ref) + else + rho(is:ie, js:je, ks:ke) = this%density_elem(T(is:ie, js:je, ks:ke), & + S(is:ie, js:je, ks:ke), pressure(is:ie, js:je, ks:ke)) + endif +end subroutine a_calculate_density_array_3d +``` + +This is a **whole-array assignment invoking an elemental type-bound procedure through a polymorphic +`this`** — the compiler must resolve, per call, which concrete override of `density_elem`/ +`density_anomaly_elem` to invoke (v-table dispatch), and then apply it across an array section. This +is the fallback path used by **every EOS form that does not supply its own array/2d/3d override** +(see §3): `linear`, `UNESCO`, `Jackett06`, `TEOS10`, `Wright_full`, `Wright_red`, `Roquet_SpV`. + +### 1.3 Concrete class selection: `select case` / `allocate` at init + +The concrete class is chosen once, at `EOS_init`/`EOS_manual_init` time, with an ordinary +`select case` driving `allocate( :: EOS%type)`: + +```fortran +! MOM_EOS.F90:2122-2148 +if (allocated(EOS%type)) deallocate(EOS%type) ! Needed during testing which re-initializes +select case (EOS%form_of_EOS) + case (EOS_LINEAR) + allocate(linear_EOS :: EOS%type) + case (EOS_UNESCO) + allocate(UNESCO_EOS :: EOS%type) + case (EOS_WRIGHT) + allocate(buggy_Wright_EOS :: EOS%type) + case (EOS_WRIGHT_FULL) + allocate(Wright_full_EOS :: EOS%type) + case (EOS_WRIGHT_REDUCED) + allocate(Wright_red_EOS :: EOS%type) + case (EOS_JACKETT06) + allocate(Jackett06_EOS :: EOS%type) + case (EOS_TEOS10) + allocate(TEOS10_EOS :: EOS%type) + case (EOS_ROQUET_RHO) + allocate(Roquet_rho_EOS :: EOS%type) + case (EOS_ROQUET_SPV) + allocate(Roquet_SpV_EOS :: EOS%type) +end select +select type (t => EOS%type) + type is (linear_EOS) + call t%set_params_linear(Rho_T0_S0, dRho_dT, dRho_dS, dRho_dp) + type is (buggy_Wright_EOS) + call t%set_params_buggy_Wright(use_Wright_2nd_deriv_bug) +end select +``` + +Every call site downstream (e.g. `calculate_density_3d`, `MOM_EOS.F90:427`) then dispatches through +`EOS%type%calculate_density_array_3d(...)` — ordinary Fortran type-bound-procedure dispatch on a +polymorphic `allocatable` component. + +### 1.4 Why this is fatal on device + +Two distinct, stacked problems: + +1. **The outer v-table dispatch itself.** `EOS%type%calculate_density_array_3d(...)` is a runtime + (dynamic) dispatch — the compiler must consult a type descriptor attached to the allocatable + polymorphic component to find the correct concrete procedure. This dispatch happens once per host + call (not per grid point), so it is not the primary GPU blocker by itself — but it means the + *body* of whatever gets called still carries `class(EOS_base)/class(), intent(in) :: this` + as a live dummy argument in its interface. +2. **The polymorphic `this` inside device loops.** Any elemental procedure bound through `this` + (`this%density_elem(...)`, `this%density_anomaly_elem(...)`, or even a same-module wrapper that + takes `this` as its first dummy) that gets called *inside* a `do concurrent`/OpenMP-target region + forces nvfortran to attempt to pass/copy the polymorphic descriptor into the device region per + iteration. In practice this either (a) silently produces an **implicit copy of `this` that cannot + be prevented** (documented in-source, see §2.3), which is a correctness/perf hazard, or (b) throws + an outright **runtime error on GPU with nvfortran** for the forms that hadn't yet been rewritten + (the exact phrase used repeatedly in-source, e.g. `MOM_EOS_Wright.F90:108-109`, + `MOM_EOS_Roquet_rho.F90:260-261`). Passing `class(*)`/polymorphic `this` into a `do concurrent` or + `target` region is exactly the failure mode called out in `00-architecture.md` guiding principle 4 + and quick-reference item in §9 ("**Never** pass `class(*)`/polymorphic `this`"). + +Net effect: the seven EOS forms that still rely purely on the `EOS_base` fallbacks (§1.2) cannot be +called from inside a device array loop at all without hitting this. The two forms that needed +GPU-resident 2D/3D density evaluation (Wright and Roquet_rho, used by the ported +`MOM_PressureForce_FV.F90`/`MOM_density_integrals.F90` pressure-gradient path) had to be rewritten. + +--- + +## 2. The new strategy: free `_loc` kernels + explicit `do concurrent` overrides + +### 2.1 Pattern + +For each rewritten form, every elemental kernel that used to be a type-bound procedure +`foo_elem_XXX(this, T, S, p, ...)` is **duplicated** as a free (non-type-bound) elemental function +`foo_elem_XXX_loc(T, S, p, ...)` with **no `this` parameter at all**, and the original +`this`-taking procedure becomes a **thin one-line wrapper** that just calls the `_loc` version (kept +only so the deferred-interface contract in `EOS_base` is still satisfied for scalar/1-D call sites +that go through the ordinary polymorphic path). New **direct** `calculate_density_array_2d/3d`, +`calculate_density_derivs_2d/3d`, and (for Roquet_rho) `calculate_density_second_derivs_2d` +overrides are added on the concrete type, each containing an explicit `do concurrent (k,j,i)` / +`do concurrent (j,i)` loop that calls the `_loc` kernel directly — never through `this`. + +### 2.2 Quoted example — Wright `_loc`/wrapper pair (`MOM_EOS_Wright.F90:90-117`) + +```fortran +real elemental function density_elem_buggy_Wright_loc(T, S, pressure) + real, intent(in) :: T !< potential temperature relative to the surface [degC]. + real, intent(in) :: S !< salinity [PSU]. + real, intent(in) :: pressure !< pressure [Pa]. + ! Local variables + real :: al0, p0, lambda + al0 = (a0 + a1*T) +a2*S + p0 = (b0 + b4*S) + T * (b1 + T*(b2 + b3*T) + b5*S) + lambda = (c0 +c4*S) + T * (c1 + T*(c2 + c3*T) + c5*S) + density_elem_buggy_Wright_loc = (pressure + p0) / (lambda + al0*(pressure + p0)) +end function density_elem_buggy_Wright_loc + +!> Wrapper for density_elem_buggy_Wright_loc created to preserve API while calling +!! density_elem_buggy_Wright without "this" variable that causes runtime errors on +!! gpu runs with nvfortran. +real elemental function density_elem_buggy_Wright(this, T, S, pressure) + class(buggy_Wright_EOS), intent(in) :: this !< This EOS + real, intent(in) :: T, S, pressure + density_elem_buggy_Wright = density_elem_buggy_Wright_loc(T, S, pressure) +end function density_elem_buggy_Wright +``` + +This is the literal pattern introduced by commit `52a1b3954` ("Added local versions of +density_elem and density_derivs without 'this' argument"). Its diff shows the **before** state was +exactly the disaster case described in §1.4: the 2D override itself used to call the `this`-taking +form *inside* the `do concurrent`: + +```diff + else + do concurrent (j=js:je, i=is:ie) +- rho(i,j) = density_elem_buggy_Wright(this, T(i,j), S(i,j), pressure(i,j)) ++ rho(i,j) = density_elem_buggy_Wright_loc( T(i,j), S(i,j), pressure(i,j)) + enddo +``` +(commit `52a1b3954`, `MOM_EOS_Wright.F90`) + +### 2.3 Quoted example — 3D override calling the `_loc` kernel (`MOM_EOS_Wright.F90:1024-1061`) + +```fortran +subroutine calculate_density_array_3d_buggy_Wright(this, T, S, pressure, rho, dom, rho_ref) + class(buggy_Wright_EOS), intent(in) :: this + ... + ! NOTE: There is an implicit copy of `this` which cannot yet be prevented. + ! Possibly because Nvidia cannot associate `this` with `EOS%type`. + if (present(rho_ref)) then + do concurrent (k=ks:ke, j=js:je, i=is:ie) + rho(i,j,k) = density_anomaly_elem_buggy_Wright(this, T(i,j,k), S(i,j,k), & + pressure(i,j,k), rho_ref) + enddo + else + do concurrent (k=ks:ke, j=js:je, i=is:ie) + rho(i,j,k) = density_elem_buggy_Wright_loc( T(i,j,k), S(i,j,k), pressure(i,j,k)) + enddo + endif +end subroutine calculate_density_array_3d_buggy_Wright +``` + +Two things worth flagging for the knowledge base: + +- The **plain-density branch** (`rho_ref` absent) calls the `_loc` free function — clean, no `this`. +- The **anomaly branch** (`rho_ref` present) **still calls `density_anomaly_elem_buggy_Wright(this, ...)`** + (2D at `:1013`, 3D at `:1053`, both verified) — i.e. in `MOM_EOS_Wright.F90` the `_loc` treatment + was only applied to `density_elem` and `calculate_density_derivs_elem`. There is **no** + `density_anomaly_elem_buggy_Wright_loc` free function in the file at all (grep-confirmed: the only + `_loc` kernels in Wright are `density_elem_buggy_Wright_loc` at `:90` and + `calculate_density_derivs_elem_buggy_Wright_loc` at `:199`). + +There are therefore **two structurally different residuals**, and it matters that the knowledge base +keep them apart: + + 1. **A genuinely `this`-dereferencing device loop (unfinished work).** The Wright anomaly branch + (`:1013, :1053`) calls `density_anomaly_elem_buggy_Wright(this, ...)` *inside* the `do concurrent` + body. That is not a compiler limitation — `MOM_EOS_Roquet_rho.F90` proves the fix is available: + it *does* have `density_anomaly_elem_Roquet_rho_loc` (`:274`) and its array overrides call it in + the anomaly branch (`:744, :783`). Wright simply never got a `density_anomaly` `_loc` kernel + written. This is a mechanical, finishable gap, not an nvfortran wall. + 2. **A residual `this` that appears only in the *signature*, never in the loop body (compiler + limitation).** The Wright and Roquet derivs overrides call *only* the `_loc` kernel inside the + loop (Wright `:1117, :1150`; Roquet `:820, :854`) yet still carry the + "**implicit copy of `this` which cannot yet be prevented**" comment. Here `this` is dereferenced + nowhere in the region — it survives only because the enclosing subroutine must declare + `class(_EOS), intent(in) :: this` to satisfy the `EOS_base` deferred-binding contract, + and nvfortran materializes/copies that descriptor for the device region regardless. This is the + genuinely open compiler issue. + + **Correction to an earlier draft of this doc:** it is *not* the case that Roquet's + `calculate_density_derivs_2d` "was never converted to call a `_loc` kernel." Line `:820` does call + `calculate_density_derivs_elem_Roquet_rho_loc`; the loop body never touches `this`. The retained + "implicit copy…cannot yet be prevented" note at Roquet `:817` is therefore case (2) — a genuine + compiler-limitation annotation on a fully-converted path — and its wording is simply the *older* + phrasing, left inconsistent with the newer "…called via their free-function (`_loc`) form rather + than through the polymorphic `this` binding, which causes runtime errors in `do concurrent` regions + offloaded to the GPU with nvfortran" comment used on the sibling 3D-density, 3D-derivs, and + 2D-second-derivs overrides (`:778-780, :850-852, :889-891`). Both comments describe the *same* + case-(2) residual; only the wording differs. The `MOM_EOS_Wright.F90` comment sites are + `:1008, :1048, :1114, :1147`; the sole Roquet "implicit copy" site is `:817`. + + > **Resolved (2026-07-14):** The Wright anomaly `this` branch is mainline-safe but a live hazard on + > the pf branch. On `dev/gpu` the generic 2D/3D-plus-`rho_ref` dispatch is reached only from host + > paths, so passing `this` costs nothing. On `port/pressureforce-benchmark_ALE` it *is* a live bug: + > the k-blocked `int_density_dz_generic_plm` (`MOM_density_integrals.F90`) calls 3-D + > `calculate_density(..., rho_ref=rho_ref)` with `use_rho_ref = .true.` **by default**, dispatching + > into the `present(rho_ref)` branch that passes polymorphic `this` inside a `do concurrent` + > (`calculate_density_array_2d_buggy_Wright` and its 3-D sibling). Merge gate for that branch: add + > `density_anomaly_elem_buggy_Wright_loc` first — trivial, and Roquet proves the pattern. + +### 2.4 `int_density_dz_wright`: whole-routine offload, not just the elemental kernel + +`int_density_dz_wright` (`MOM_EOS_Wright.F90:426-706`) is not called through `EOS_base` dispatch at +all (it's a free module procedure used directly by `MOM_PressureForce_FV.F90`/ +`MOM_density_integrals.F90`), so it has no `this`/polymorphism problem — but it is architecturally +part of the same EOS-layer port and was rewritten in commit `692abbc67` ("port int_density_dz_wright") +from plain nested `do`-loops to: +- `!$omp target enter data map(alloc: z0pres, al0_2d, p0_2d, lambda_2d, intz)` / matching + `exit data map(release:...)` bracketing the whole routine (`:548, :704`), +- `do concurrent (j=..., i=...)` for the pointwise vertical-integral computation (`:550-605`), +- `!$omp target teams loop collapse(2) private(...)` (long private list: `hWght, hL, hR, iDenom, + hWt_LL, hWt_LR, hWt_RR, hWt_RL, m, wt_L, wt_R, wtT_L, wtT_R, al0, p0, lambda, dz, p_ave, I_al0, + I_Lzz, eps, eps2, intz`) for the horizontal (Boole's-rule) integrals in x and y (`:608-654, + 657-703`), replacing the old collapsed `do j=... ; do I=...` form. `target teams loop` is used + here (not `do concurrent`) because the horizontal loops carry a serial inner `do m=2,4` and a + sizeable private scalar/array list per (i,j) — the same pattern vertical friction uses for its + tridiagonal column solve (see `00-architecture.md` §4.3). + +--- + +## 3. Per-form port status + +Objective evidence: count of `do concurrent` occurrences per EOS source file (a form with array/2d/3d +overrides has them; a form still on the `EOS_base` fallback path has zero, because its only elemental +kernels are plain scalar `elemental function`s with no device directives of their own): + +| File | `do concurrent` count | Has `_loc` kernels? | Overrides `array_2d/3d`, `derivs_2d/3d`? | Overrides `second_derivs_2d`? | Status | +|---|---|---|---|---|---| +| `MOM_EOS_Wright.F90` (`buggy_Wright_EOS`) | 9 | yes (partial: `density_elem`, `derivs_elem` only) | yes | **no** | **Ported** (density + 1st derivs 2D/3D); anomaly branch still passes `this` | +| `MOM_EOS_Roquet_rho.F90` (`Roquet_rho_EOS`) | 11 | yes (density, anomaly, derivs, 2nd-derivs) | yes | **yes** | **Ported**, most complete conversion | +| `MOM_EOS_linear.F90` (`linear_EOS`) | 0 | no | no | no | Polymorphic fallback only | +| `MOM_EOS_UNESCO.F90` (`UNESCO_EOS`) | 0 | no | no | no | Polymorphic fallback only | +| `MOM_EOS_Jackett06.F90` (`Jackett06_EOS`) | 0 | no | no | no | Polymorphic fallback only | +| `MOM_EOS_TEOS10.F90` (`TEOS10_EOS`) | 0 | no | no | no | Polymorphic fallback only | +| `MOM_EOS_Wright_full.F90` (`Wright_full_EOS`) | 0 | no | no | no | Polymorphic fallback only | +| `MOM_EOS_Wright_red.F90` (`Wright_red_EOS`) | 0 | no | no | no | Polymorphic fallback only | +| `MOM_EOS_Roquet_SpV.F90` (`Roquet_SpV_EOS`) | 0 | no | no | no | Polymorphic fallback only | + +Confirmed by `grep -c 'type, extends\|procedure ::.*array_2d\|...' `: each of the seven unported +files' `type, extends (EOS_base) :: _EOS ... end type` block contains **no** override of +`calculate_density_array_2d/3d`, `calculate_density_derivs_2d/3d`, or +`calculate_density_second_derivs_2d` — every one of them relies purely on the `EOS_base` fallbacks +in §1.2, i.e. on `this%density_elem(...)` applied elementally across whole array sections. Any code +path that ends up calling `calculate_density_3d`/`calculate_density_derivs_3d` for `EOS_LINEAR`, +`EOS_UNESCO`, `EOS_JACKETT06`, `EOS_TEOS10`, `EOS_WRIGHT_FULL`, `EOS_WRIGHT_REDUCED`, or +`EOS_ROQUET_SPV` still routes through the polymorphic `this%..._elem` dispatch and is the disaster +case from §1.4 if it is ever invoked from a device loop. `EOS_DEFAULT` in `MOM_EOS.F90:192` is +`EOS_WRIGHT_FULL_STRING` — i.e. **the model's default EOS is still on the unported path**; only +configurations that explicitly select `WRIGHT` (buggy) or `ROQUET_RHO` get the GPU-safe direct +kernels. + +--- + +## 4. `calculate_density_*` generic wrappers: fast path + rescale path (`MOM_EOS.F90`) + +`MOM_EOS.F90` is the generic front door (`interface calculate_density`, `:70-78`; +`calculate_density_derivs`, `:87-92`; `calculate_density_second_derivs`, `:101-104`). Every 2D/3D +generic wrapper (`calculate_density_2d`, `calculate_density_3d` at `:427`, +`calculate_density_derivs_2d`, `calculate_density_derivs_3d` at `:1055`, +`calculate_density_second_derivs_2d` at `:1236`) has the identical shape: test whether the EOS's +unit-rescaling factors are all exactly `1.0` and, if so, skip the rescale copies and call +`EOS%type%calculate_..._2d/3d` directly on the caller's arrays; otherwise rescale `T`/`S`/`pressure` +into temporaries first. E.g. `calculate_density_3d` (`MOM_EOS.F90:427-485`): + +```fortran +if ((EOS%RL2_T2_to_Pa == 1.0) .and. (EOS%R_to_kg_m3 == 1.0) .and. & + (EOS%C_to_degC == 1.0) .and. (EOS%S_to_ppt == 1.0)) then + call EOS%type%calculate_density_array_3d(T, S, pressure, rho, domain, rho_ref=rho_ref) +else ! This is the same as above, but with some extra work to rescale variables. + pres(is:ie, js:je, ks:ke) = EOS%RL2_T2_to_Pa * pressure(is:ie, js:je, ks:ke) + Ta(is:ie, js:je, ks:ke) = EOS%C_to_degC * T(is:ie, js:je, ks:ke) + Sa(is:ie, js:je, ks:ke) = EOS%S_to_ppt * S(is:ie, js:je, ks:ke) + if (present(rho_ref)) then + call EOS%type%calculate_density_array_3d(Ta, Sa, pres, rho, domain, rho_ref=EOS%R_to_kg_m3*rho_ref) + else + call EOS%type%calculate_density_array_3d(Ta, Sa, pres, rho, domain) + endif +endif +``` + +This fast path exists purely for the common non-dimensional-testing configuration (scale factors +== 1) and avoids allocating/filling three full-size rescale temporaries (`pres`, `Ta`, `Sa`) per +call — a performance optimization orthogonal to the polymorphism problem, but relevant because it +determines whether the call into `EOS%type%calculate_density_array_3d` receives the caller's own +arrays directly (fast path) or freshly-computed local temporaries (rescale path); either way the +call itself still goes through the same `EOS%type%...` dynamic dispatch described in §1.3. + +The dispatch-level guard everywhere upstream of the fast path is `if (.not. allocated(EOS%type)) +call MOM_error(FATAL, ...)` (e.g. `:1034, :1096, :1184`) — a defensive check that the concrete class +was actually allocated by `EOS_init`/`EOS_manual_init` before any dispatch is attempted. + +--- + +## 5. `int_density_dz_*` / `int_spec_vol_dp_*` pressure-integral routines + +These are free module procedures (not `EOS_base` type-bound), called directly from +`MOM_PressureForce_FV.F90` for the two EOS-specific fast paths (Wright, linear) and from +`src/core/MOM_density_integrals.F90`'s **generic** PCM/PLM/PPM routines +(`int_density_dz_generic_pcm/plm/ppm`, `int_spec_vol_dp_generic_pcm/plm`) for every other EOS form +— the generic routines call back into the polymorphic `calculate_density`/`calculate_density_derivs` +generic interface, so they inherit whichever port status the underlying EOS form has (§3). + +Port status by direct grep for `do concurrent` in each EOS-specific integral file: + +| Routine | File | Ported? | +|---|---|---| +| `int_density_dz_wright` | `MOM_EOS_Wright.F90:426` | **Yes** — `do concurrent` (pointwise) + `!$omp target teams loop collapse(2)` (Boole's-rule horizontal integrals), commit `692abbc67` | +| `int_spec_vol_dp_wright` | `MOM_EOS_Wright.F90:713` (same file, later section) | **No** — plain nested loops, zero directives in this region | +| `int_density_dz_linear` / `int_spec_vol_dp_linear` | `MOM_EOS_linear.F90:277,483` | No | +| `int_density_dz_wright_full` / `int_spec_vol_dp_wright_full` | `MOM_EOS_Wright_full.F90:397,669` | No | +| `int_density_dz_wright_red` / `int_spec_vol_dp_wright_red` | `MOM_EOS_Wright_red.F90:399,671` | No | +| Roquet_rho | — | Has **no** `int_density_dz`/`int_spec_vol_dp` of its own; always routes through `int_density_dz_generic_*` in `MOM_density_integrals.F90`, which calls the ported `calculate_density_3d`/`calculate_density_derivs_3d` array entry points for Roquet_rho specifically | + +So within `MOM_EOS_Wright.F90` itself the port is **half-done**: the density-anomaly integral +(`int_density_dz_wright`) is GPU-resident, but the companion specific-volume integral +(`int_spec_vol_dp_wright`, used for the non-Boussinesq path) is not. + +### `port/pressureforce-benchmark_ALE` — k-blocking `int_density_dz_generic_plm` + +This branch (commits `8c6881e6c` "move kblock inside int_density_dz_generic_plm", `1038a4921`, +`a3e889601`, on top of merged commit `c82e1254a` "vertvisc: Fix CS memory management") k-blocks the +**generic** ALE/PLM pressure-integral path in `src/core/MOM_density_integrals.F90` and its submodule +implementation `MOM_density_integrals_s.F90` — a different track from the EOS-layer `_loc` rewrite. +Signature change: `int_density_dz_generic_plm(k, ...)` → `int_density_dz_generic_plm(kstart, kend, +...)`, with `dpa`/`intz_dpa`/`intx_dpa`/`inty_dpa` gaining a `SZK_(GV)` third dimension so a whole +k-range can be processed per call. Inside `generic_plm_update_dpa`, the k-loop is folded into the +existing tiled `do concurrent`: + +```fortran +! before (single k, passed in from caller) +do concurrent (j=jstart:jend, i=istart:iend) + ... +enddo +! after (k-blocked) +do concurrent (k=kstart:kend, j=jstart:jend, i=istart:iend) + ii = i-istart+1 ; jj = j-jstart+1 + dz = e(i,j,K) - e(i,j,K+1) + ... +enddo +``` + +but the **EOS evaluation itself inside that k-loop still calls the generic `calculate_density` +interface** (`call calculate_density(T5, S5, p5, T25, TS5, S25, r5, EOS, EOSdom_h5, rho_ref=rho_ref)`, +wrapped in its own `do k=kstart,kend` loop, separate from the k-blocked `do concurrent`) — i.e. this +branch k-blocks the *driver* loop structure but does not touch or depend on which EOS form is +active; it inherits the polymorphic-dispatch status of whatever `EOS%type` is allocated, same as the +unmodified generic PCM/PPM routines. This confirms the k-blocking effort (§5 of +`00-architecture.md`) and the EOS `_loc`-rewrite effort are two independent, not-yet-unified tracks. + +Branches `remotes/origin/eos-3d`, `remotes/origin/efficient_density_integrals_new_api`, +`efficient_density_integrals_rebase`, `efficient_density_integrals_stanley` are earlier/alternative +staging points for the same `calculate_density_3d`/`_derivs_3d`/`_second_derivs_2d` API additions +that commit `7c7af5572` ultimately merged (its own commit message: "These were extracted from a +larger pull request supporting pressure density integrals (#156)" — i.e. `#185`/`7c7af5572` is a +narrower cherry-pick of that larger, still-unmerged effort). + +--- + +## 6. What remains, boilerplate cost, and the endgame + +### 6.1 What remains + +- **Seven of nine EOS forms** (`linear`, `UNESCO`, `Jackett06`, `TEOS10`, `Wright_full`, + `Wright_red`, `Roquet_SpV`) have **no** GPU-safe 2D/3D density/derivs path at all — including + `Wright_full`, which is `EOS_DEFAULT` (`MOM_EOS.F90:192`). Any GPU run using the default EOS (or + any of the other six) that reaches a 3D array density/derivs call falls straight back onto + `EOS_base`'s `this%..._elem` array-section fallback (§1.2), the exact pattern documented as fatal. +- Even within the two "ported" forms, the conversion is **incomplete**: `buggy_Wright_EOS` has no + `calculate_density_second_derivs_2d` override (falls back to `EOS_base`'s), and its + `density_anomaly_elem` path still passes `this` inside `do concurrent` (§2.3). `int_spec_vol_dp_wright` + (companion to the ported `int_density_dz_wright`) is unported. +- The "implicit copy of `this` which cannot yet be prevented" comment is left **unresolved** in + five places (`MOM_EOS_Wright.F90:1008,1048,1114,1147`; `MOM_EOS_Roquet_rho.F90:817`). As split in + §2.3, these are **not all the same thing**: + - `MOM_EOS_Wright.F90:1114,1147` (derivs 2D/3D) and `MOM_EOS_Roquet_rho.F90:817` (derivs 2D) sit + above `do concurrent` bodies that call **only** the `_loc` kernel and never touch `this` — these + are the pure **compiler-limitation** case: `this` survives only in the formal parameter list + (mandated by the `EOS_base` deferred-binding contract) and nvfortran's do-concurrent/target + lowering materializes/copies that descriptor regardless. No workaround for this specific residual + is recorded in-source yet. + - `MOM_EOS_Wright.F90:1008,1048` head `calculate_density_array_2d/3d_buggy_Wright`, whose **anomaly + branch** (`:1013,:1053`) genuinely dereferences `this` inside the loop. That part is + **unfinished work** (no `density_anomaly_elem_buggy_Wright_loc` exists), finishable by copying + Roquet's approach — see §2.3. + +### 6.2 Boilerplate cost of the `_loc` duplication, per form + +For a form fully converted like `Roquet_rho`, the pattern requires, per elemental kernel that needs +device access: +1. A new free `_loc` elemental function/subroutine with the same body but no `this` (full + duplication of the arithmetic — not a refactor, a copy). +2. The original `this`-taking type-bound procedure reduced to a one-line wrapper calling the `_loc` + version (kept only to satisfy `EOS_base`'s deferred interface for scalar/1-D generic callers). +3. A new `calculate_density_array_2d`/`_3d`, `calculate_density_derivs_2d`/`_3d`, and (optionally) + `calculate_density_second_derivs_2d` override on the concrete type, each hand-written with its own + `do concurrent` loop and `dom(...)` index bookkeeping — duplicating the loop-and-index-slicing + logic that `EOS_base`'s fallback already provides generically for free. + +Concretely: `MOM_EOS_Roquet_rho.F90` grew from a file with only elemental kernels to one with **11 +distinct `do concurrent` loops**, each hand-duplicating the domain-slicing arithmetic already present +once in `MOM_EOS_base_type.F90`'s generic fallbacks. Commit `7c7af5572` alone added +174 lines to +`MOM_EOS_Roquet_rho.F90` and +77 to `MOM_EOS_Wright.F90` purely for the 2D/3D array overrides (plus ++92 to `MOM_EOS_base_type.F90` for the generic fallback siblings that non-ported forms still use). +Scaling this same treatment to the remaining seven forms would mean **7× more duplicated kernels** +(each EOS form already has ~7-9 elemental kernels: density, anomaly, spec_vol, spec_vol anomaly, +derivs, second-derivs, specvol-derivs, compress) plus 7×3-5 hand-written array-loop overrides — a +large, mechanical, error-prone amount of copy-paste, and every future bugfix to an elemental kernel's +math must now be applied in two places (the `this`-taking original and the `_loc` copy) unless the +original is reduced to a pure pass-through (as done for the fully-converted kernels). + +### 6.3 Prescriptive recipe: port one more EOS form, bit-for-bit + +This is the exact, mechanical procedure to give any remaining form (`linear`, `UNESCO`, `Jackett06`, +`TEOS10`, `Wright_full`, `Wright_red`, `Roquet_SpV`) a GPU-safe 2D/3D path, matching the merged +`buggy_Wright`/`Roquet_rho` pattern. It touches **exactly one file**, `MOM_EOS_
.F90` — no +change to `MOM_EOS.F90` is needed, because the generic front-door wrappers there already dispatch to +`EOS%type%calculate_density_array_2d/3d`, `..._derivs_2d/3d`, `..._second_derivs_2d` (§4); overriding +those bindings on the concrete type is automatically picked up. Use `MOM_EOS_Roquet_rho.F90` as the +reference implementation (the most complete conversion). + +**Step 0 — scope.** Decide which of the five array entry points the GPU path actually needs. The full +set is `calculate_density_array_2d`, `calculate_density_array_3d`, `calculate_density_derivs_2d`, +`calculate_density_derivs_3d`, `calculate_density_second_derivs_2d`. Roquet_rho overrides all five; +buggy_Wright overrides the first four (no `second_derivs_2d`). Each override you add shadows the +`EOS_base` `a_*` fallback **for that form only**; forms you don't touch keep using the fallback. + +**Step 1 — `_loc` free kernels (one per elemental kernel the overrides call).** For each of +`density_elem`, `density_anomaly_elem`, `calculate_density_derivs_elem`, and (only if doing +second_derivs) `calculate_density_second_derivs_elem`, add a free (non-type-bound) elemental +procedure `__loc(T, S, pressure[, ref][, out args])` whose body is **copied verbatim** +from the existing `_` with the `this` dummy deleted. Do **not** refactor or re-parenthesize +the arithmetic — bitwise reproducibility (`00-architecture.md` principle 2) requires identical +floating-point operation order. Reference bodies: Roquet `density_elem_Roquet_rho_loc` (`:204`), +`density_anomaly_elem_Roquet_rho_loc` (`:274`), `calculate_density_derivs_elem_Roquet_rho_loc` +(`:375`), `calculate_density_second_derivs_elem_Roquet_rho_loc` (`:466`). **Do not skip +`density_anomaly` — that omission is exactly the unfinished-work gap in Wright (§2.3).** + +**Step 2 — reduce each original kernel to a one-line wrapper.** Keep the +`class(_EOS), intent(in) :: this` dummy and the `!>` doc comment (the deferred `EOS_base` +interface still requires the binding for scalar/1-D generic callers), replace the body with a single +call to the `_loc` version. Reference: Roquet `density_elem_Roquet_rho` (`:262-268`), +`density_anomaly_elem_Roquet_rho` (`:335-342`). + +**Step 3 — declare the array overrides in the type block.** In `type, extends(EOS_base) :: _EOS +… contains`, add (mirroring Roquet `:185-195`): +```fortran +procedure :: calculate_density_array_2d => calculate_density_array_2d_ +procedure :: calculate_density_array_3d => calculate_density_array_3d_ +procedure :: calculate_density_derivs_2d => calculate_density_derivs_2d_ +procedure :: calculate_density_derivs_3d => calculate_density_derivs_3d_ +procedure :: calculate_density_second_derivs_2d => calculate_density_second_derivs_2d_ ! optional +``` + +**Step 4 — implement each override.** Copy the **signature** of the matching `EOS_base` fallback +(`MOM_EOS_base_type.F90`: `a_calculate_density_array_2d` `:268`, `_array_3d` `:298`, +`a_calculate_density_derivs_2d` `:428`, `_derivs_3d` `:456`, `a_calculate_density_second_derivs_2d` +`:542`). Body: unpack `dom(rank,2)` into `is/ie, js/je[, ks/ke]`, then write an explicit +`do concurrent (…, j=js:je, i=is:ie)` that calls the `_loc` kernel. For the density arrays, +write **both** branches — `present(rho_ref)` → `density_anomaly_elem__loc(…, rho_ref)`, else +`density_elem__loc(…)`. Reference: Roquet `calculate_density_array_3d_Roquet_rho` (`:782-789`), +`calculate_density_derivs_3d_Roquet_rho` (`:853-855`), `calculate_density_second_derivs_2d_Roquet_rho` +(`:892-893`). + +**Step 5 — annotate.** Add the NOTE comment above the loop; prefer the newer, accurate wording +(Roquet `:778-780`) over the legacy "implicit copy" phrasing. Leave `this` in the signature +(unavoidable — see §2.3 case 2). + +**Step 6 — self-check (no build).** `grep -c 'do concurrent' MOM_EOS_.F90` should rise by the +number of loops added (Wright=9, Roquet=11 for reference). Confirm **no** `_EOS` array override +calls `_(this, …)` inside a `do concurrent` — every device-region call must target a +`_loc` kernel. The build+numerical check (`MOM_checksums` hchksum CPU vs GPU, `00-architecture.md` +§9) is the acceptance gate but is out of scope for a source-only agent. + +**Out of scope for this recipe:** the `int_density_dz_` / `int_spec_vol_dp_` pressure +integrals (§2.4, §5) are a *separate* whole-routine offload (`enter data` + `do concurrent` + +`target teams loop`), independent of the `_loc` dispatch rewrite. Only `int_density_dz_wright` is +done; `int_spec_vol_dp_wright` and every other form's integrals are not. + +### 6.4 Proposal (not existing code): a polymorphism-free end-state + +> **Everything in this subsection is a *proposal* sketched from the evidence, not code present in the +> tree.** It is offered so a future agent has a target architecture; verify feasibility before acting. + +The `_loc` recipe (§6.3) removes `this` from the *loop body* but cannot remove it from the override +*signature* — the residual descriptor copy (§2.3 case 2) persists because `EOS_base`'s deferred +bindings mandate the `this` dummy. A structural fix would drop the `class(EOS_base), allocatable :: +type` component (`MOM_EOS.F90:162`) and the abstract type entirely, replacing runtime v-table +dispatch with a compile-time `select case (EOS%form_of_EOS)` at the generic front door: + +- The elemental `_loc` kernels are **already `this`-free**, so they carry over unchanged and become + plain public module procedures of each `MOM_EOS_` module. +- The generic wrappers in `MOM_EOS.F90` (`calculate_density_3d` etc., §4) replace + `call EOS%type%calculate_density_array_3d(…)` with `select case (EOS%form_of_EOS) ; case + (EOS_WRIGHT) ; call calculate_density_array_3d_buggy_Wright(…) ; case (EOS_ROQUET_RHO) ; … ; end + select`, calling the form's array routine as a module procedure with **no `this`**. This eliminates + both the v-table dispatch (§1.4 problem 1) and the descriptor-copy residual (§1.4 problem 2) + structurally, in one place, rather than per kernel. +- Instance parameters move onto `EOS_type` directly (a small non-polymorphic surface: `buggy_Wright` + needs only `three` + `use_Wright_2nd_deriv_bug`; `linear` needs `Rho_T0_S0, dRho_dT, dRho_dS, + dRho_dp`; most forms carry only module `parameter`s already, e.g. Wright's `a0…c5`). `set_params` + fills the relevant fields; the existing `allocate( :: EOS%type)` + `select type` + (`:2123`) is deleted. +- Trade-off: the `select case` must enumerate every form at each of the ~5 array entry points (a + fixed, bounded amount of code), versus today's open-ended per-form `_loc` duplication. It also + couples `MOM_EOS.F90` to every form module (already effectively true via `use`). The upstream + `eos-3d` / `efficient_density_integrals_*` branches and PR `#156` are the place to check whether a + variant of this is already in progress before re-deriving it. + +### 6.5 Is there a plan to eliminate polymorphism entirely? + +**No integration/removal plan is recorded in-source.** What exists today is a per-form, per-kernel, +opt-in escape hatch (`_loc` + explicit override) applied to exactly the two forms actually exercised +by the ported pressure-force path (`buggy_Wright_EOS`, `Roquet_rho_EOS`), while `EOS_base` itself +(the abstract type, its `deferred`/polymorphic interface, and its `this`-based fallbacks) is +untouched and still the only implementation for seven forms including the default. There is no +in-repo comment, TODO, or branch proposing to replace `EOS_type`'s `class(EOS_base), allocatable` +component with a non-polymorphic tagged-union / `select case (form_of_EOS)` dispatch at the call +site (which would sidestep the v-table/`this`-descriptor problem structurally rather than by +per-kernel duplication) — the `_loc` pattern as merged is a **local workaround**, not the +architectural fix `00-architecture.md` alludes to when it says "the EOS layer is being rewritten to +avoid it." Based on the commit history (`7c7af5572`/`#185` explicitly extracted from a larger, +still-unmerged PR `#156`, plus the parallel `eos-3d`/`efficient_density_integrals_*` staging +branches), the wider rewrite is in progress upstream but not yet visible as a completed design in +this tree; a porting agent picking this up next should treat "convert the remaining 7 forms with the +same `_loc` boilerplate" as the known-mechanical stopgap, and treat "replace `class(EOS_base)` +dispatch with a non-polymorphic form" as the still-open architectural question. + +--- + +## Verification notes + +Opus verification pass (source + git only; no build/run). Checked every factual claim against +`MOM_EOS.F90`, `MOM_EOS_base_type.F90`, `MOM_EOS_Wright.F90`, `MOM_EOS_Roquet_rho.F90`, and commits +`7c7af5572`, `52a1b3954`, `692abbc67`, plus the named branches. + +**Confirmed (verified against code/git):** +- `do concurrent` per-form table (§3): Wright 9, Roquet_rho 11, all seven other forms 0 — exact. +- Wright anomaly branch still passes `this` inside `do concurrent`: 2D `:1013`, 3D `:1053` — verified. +- `Wright_full` is `EOS_DEFAULT` (`MOM_EOS.F90:186,192`) and has zero `do concurrent` — verified. +- `int_spec_vol_dp_wright` (`:713-957`) has **no** device directives → unported; `int_density_dz_wright` + (`:426-706`) has `enter data` (`:548`) / `exit data` (`:704`) / `do concurrent` / `target teams loop` + → ported (`692abbc67`) — both verified. +- `EOS_type` `:117`, `class(EOS_base), allocatable :: type` `:162`; `select case`/`allocate` at + `:2123` (doc said `:2122`, off by one — left as-is, immaterial); base fallbacks at + `:268/298/428/456/542`; deferred bindings and generic wrappers (`calculate_density_3d :427`, + `_derivs_3d :1055`, `_second_derivs_2d :1236`, `.not. allocated` guards) — all verified. +- Commit stats: `7c7af5572` +174 Roquet / +77 Wright / +92 base / +241 `MOM_EOS.F90`; `52a1b3954` + +35/−8 Wright; `692abbc67` — verified. `#156` extraction message — verified. +- Branches `eos-3d`, `efficient_density_integrals_{new_api,rebase,stanley}`, + `port/pressureforce-benchmark_ALE` (commit `8c6881e6c`, `int_density_dz_generic_plm(kstart,kend,…)` + signature) — all exist and match — verified. +- Roquet is the more complete conversion: `_loc` kernels for density/anomaly/derivs/2nd-derivs + (`:204/274/375/466`); array overrides call `_loc` in both branches (`:744/783/788`, derivs + `:820/854`, 2nd-derivs `:893`) — verified. + +**Corrected:** +- §2.3 previously said Roquet's `calculate_density_derivs_2d` (`:817`) "was never converted to call a + `_loc` kernel." **False** — line `:820` calls `calculate_density_derivs_elem_Roquet_rho_loc`; the + loop never touches `this`. Rewrote §2.3/§6.1 to split the two residual types: (1) a genuinely + `this`-dereferencing device loop = **unfinished work** (Wright anomaly branch, fixable because + Roquet already has `density_anomaly_elem_Roquet_rho_loc :274`); (2) a `this` that appears only in + the signature = **genuine nvfortran limitation** (Wright derivs `:1114/1147`, Roquet `:817`). The + `:817` "implicit copy" note is stale wording for case (2), not evidence of an unconverted path. + +**Enhanced:** +- Added §6.3 (fully prescriptive 6-step port recipe: exact procedures, files, signatures, self-check). +- Added §6.4 (labeled *proposal*: `select case (form_of_EOS)` polymorphism-free end-state). +- Renumbered old §6.3 → §6.5 (unchanged content). + +**Confidence:** High. All quantitative claims (counts, line numbers, commit stats, branch existence) +independently reproduced from source and git. The one substantive error was in causal reasoning, not +in the underlying line references, and has been corrected. The Wright anomaly `this` branch (§2.3) has +since been traced to its callers: host-only on `dev/gpu`, a live bug on +`port/pressureforce-benchmark_ALE`. diff --git a/knowledge/gpu-knowledge/07-reproducibility.md b/knowledge/gpu-knowledge/07-reproducibility.md new file mode 100644 index 0000000..156a3a0 --- /dev/null +++ b/knowledge/gpu-knowledge/07-reproducibility.md @@ -0,0 +1,679 @@ +# Bitwise Reproducibility on `dev/gpu` + +> Drills into §0.2 and §7.2 of `00-architecture.md`. Covers the Extended Fixed Point (EFP) +> reproducing-sum machinery in `src/framework/MOM_coms.F90`, the GPU block-based restructuring +> (commit `8593a732a`), the `fix/nan_repro_sum` bugfix branch and why it could not stay `pure`, +> where reproducing sums are actually invoked in the timestep, and how `MOM_checksums.F90` verifies +> a port bit-for-bit. Source + git only; nothing here was built or run. + +--- + +## 1. The EFP fixed-point algorithm + +**Why floating-point sums aren't reproducible at all.** IEEE addition is not associative: +`(a+b)+c /= a+(b+c)` in general due to rounding. A naive parallel/distributed sum's result therefore +depends on the order values are added, which depends on domain decomposition (PE count, tile shape) +and, on GPU, thread/warp scheduling. MOM6's global diagnostics (total mass, KE, PE, heat/salt +budgets) must be identical across PE counts and across CPU/GPU builds for restart and regression +testing to mean anything, so MOM6 replaces the FP sum with an **exact integer sum**, described in +Hallberg & Adcroft 2014 (*Parallel Computing* 40(5-6), doi:10.1016/j.parco.2014.04.007; cited at +`MOM_coms.F90:101`). + +**Parameters** (`src/framework/MOM_coms.F90:31-67`): + +```f90 +integer, parameter :: accum_width = digits(1_int64) ! :31 -- 63 usable bits (excl. sign) of an int64 +integer, parameter :: prec_width = 46 ! :33 -- bits of precision per EFP "digit" +integer, parameter :: guard_width = accum_width - prec_width ! :35 -- 17 guard/carry bits +! A sum of N points does N - 1 additions, which at most adds N - 1 carry bits. +! For G guard bits, the maximum value is 2**G - 1. A summation of N values +! therefore requires that N - 1 <= 2**G - 1, or simply N <= 2**G. +integer, parameter :: max_summands = 2**guard_width ! :42 -- 2**17 = 131072 +integer(kind=int64), parameter :: prec = (2_int64)**prec_width ! :46 -- 2**46, the EFP "digit base" +integer, parameter :: efp_digits = 6 ! :54 -- number of base-2**46 words +``` + +**Decomposition.** Every `real` is decomposed into `efp_digits = 6` signed `int64` words, each +representing a base-`2^46` "digit" (`EFP_type`, `:103`): + +```f90 +type, public :: EFP_type ; private + integer(kind=int64), dimension(efp_digits) :: v !< The value in this type +end type EFP_type +``` + +The decomposition (`efp_decompose`, `:778-821`) peels off successive base-`prec` digits, most +significant first, exactly like writing a number in a mixed-radix positional system: + +```f90 +do n=1,efp_digits + ival = int(rs * I_pr(n), kind=int64) + rs = rs - ival * pr(n) + e(n) = sgn * ival +enddo +``` + +where `pr = [r_prec**2, r_prec, 1., r_prec**(-1), r_prec**(-2), r_prec**(-3)]` (`:56-57`, with +`r_prec = 2.**prec_width` the *real* value of `prec`, `:49`) — i.e. digit 3 holds the +"integer part" scale, digits 1-2 hold larger magnitudes (up to `max_efp_float = pr(1)*huge(1_int64)`, +`:64`), and digits 4-6 hold successively finer fractional remainders. Because `int()` truncation and +subtraction are exact for values representable at that scale, **the decomposition of one `real` into +6 `int64`s is exact** (no rounding is introduced beyond the fixed truncation to `prec_width` bits per +digit, which is the deliberate finite precision of the scheme, not an order-dependent error). + +**Guard bits / max_summands carry budget.** Each digit is stored in a 63-bit-capacity `int64`, but +only the low 46 bits (`prec_width`) are "normalized" content — the high 17 bits (`guard_width`) are +headroom for carry accumulation. Summing `N` per-element digit values can overflow a single digit by +at most `N-1` (one carry unit per addition), so as long as `N <= 2**guard_width = max_summands` +(131072), the accumulated carry cannot overflow the `int64` container before it is redistributed +downward into the next-more-significant digit by `carry_overflow` (`:825-845`). This is the exact +comment at `MOM_coms.F90:38-40`, quoted above. + +**Exact integer summation.** Once every real is an array of 6 `int64`s, summing many reals reduces to +6 independent columns of **exact integer addition**. Integer addition on a fixed-width machine word is +associative and commutative up to overflow (`a+b+c` gives the same bit pattern in any grouping/order, +provided no intermediate overflows) — unlike IEEE float addition, which is neither associative nor +order-invariant due to rounding. That is the entire trick: convert an inexact, order-sensitive +floating sum into an exact, order-insensitive integer sum, then convert back. + +**Reconstruction** (`ints_to_real`, `:570-579`): + +```f90 +function ints_to_real(ints) result(r) + integer(kind=int64), dimension(efp_digits), intent(in) :: ints + real :: r + integer :: i + r = 0.0 + do i=1,efp_digits ; r = r + pr(i)*ints(i) ; enddo +end function ints_to_real +``` + +`EFP_to_real` (`:937-943`) is a thin wrapper, but note it first calls `regularize_ints(EFP1%v)` +(`:941`) — which carries overflow *and* forces every digit to the sign of the overall value +(`regularize_ints`, `:849-887`) — **before** `ints_to_real(EFP1%v)` (this is why `EFP1` is +`intent(inout)`: it is normalized in place). Reconstruction sums only 6 +terms of geometrically separated magnitude (`pr(i)` spans `prec**2` down to `prec**-3`), so this final +FP summation step is itself insensitive to evaluation order in practice (fixed 6-term unrolled loop — +same order every time, on every platform, by construction) and is *not* where reproducibility would be +at risk even if it were reordered, because the loop is always a straight-line 6-iteration unroll, +never parallelized or reduced. + +**Why this is order/PE-count independent.** The full pipeline is: decompose each element to 6 exact +integers -> sum the integer columns (associative, exact) -> redistribute carries (exact, deterministic +given the summed magnitude only, not the summation order) -> reconstruct one `real` from the 6 final +digits (fixed unrolled order). Nowhere in this pipeline does the *order* in which array elements were +visited affect the final bit pattern — only the *total* per-digit integer sum matters, and integer +addition of a fixed set of addends yields a fixed total regardless of association order (so long as no +digit's running sum exceeds the guard-bit budget, which is what `max_summands`/block partitioning +guarantees, see §2). This makes the sum simultaneously: independent of loop iteration order, +independent of PE count / domain decomposition (`sum_across_PEs(ints_sum, efp_digits)`, `:226`, is +itself an exact integer all-reduce), and independent of GPU thread/team scheduling. + +--- + +## 2. GPU block restructuring: `increment_block_ints` (`MOM_coms.F90:618-772`) + +Commit `8593a732a` ("MOM_coms: GPU port of block-based repro sum", by Marshall Ward) rewrote the +inner summation loop to run as a `do concurrent` GPU reduction while still respecting the +`max_summands` carry-overflow bound, for domains of **any** size. The previous (pre-GPU) code chose +between three CPU code paths based on array size (`increment_ints_2d`, `increment_ints_faster`, +scalar `increment_ints`+`real_to_ints`) — see §3 for why one of those paths was the source of a +reproducibility bug on branch `fix/nan_repro_sum`. + +### 2.1 Partitioning math (`:665-704`) + +The compute domain (`ni x nj` elements) is split into rectangular blocks small enough that **no +single block's do-concurrent reduction can overflow the carry budget**, accounting for the two +"cumulant" additions (`block_sum -> array_sum`, `array_sum -> ints_sum`) that also consume carry +headroom: + +```f90 +max_sum_count = max_summands - 2 ! :671, reserve headroom for the 2 cumulant adds + +ni = ie - is + 1 ; nj = je - js + 1 ! :674-675, compute-domain size + +! Partition in i so that the widest i-slice fits within max_sum_count. +niblocks = (ni + max_sum_count - 1) / max_sum_count ! :678 = ceil(ni / max_sum_count) + +isize_max = (ni + niblocks - 1) / niblocks ! :686 = ceil(ni / niblocks) + +! Set jsize so that the widest i-slice times the number of j-rows does not exceed max_sum_count. +jsize = max_sum_count / isize_max ! :691 = floor(max_sum_count / isize_max) + +njblocks = (nj + jsize - 1) / jsize ! :695 = ceil(nj / jsize) + +nblocks = niblocks * njblocks ! :698 + +if (nblocks > max_sum_count) call MOM_error(FATAL, & + "reproducing sum: Number of blocks exceeds summmation carry limit.") ! :702-704 +``` + +For the default `guard_width=17` (`max_summands = 131072`), `niblocks` is "typically one" (comment +at `:681`) since 131072 columns vastly exceeds typical tile widths; the blocking logic only engages +for very large domains. The carry budget is per `increment_block_ints` *call*, and each call sums a +**single 2-D `(i,j)` slice** — `ni x nj` elements only. The vertical dimension does **not** add to a +call's carry budget: `reproducing_sum_3d` (`:353-523`) calls `increment_block_ints` once per k-layer +(`:434`) into a **separate per-layer accumulator** `ints_sums(:,k)` (`:386, 432, 435`), *not* a shared +`ints_sum`, so k never enters the `max_sum_count` arithmetic. (Correcting a natural misreading: many +k-layers do not force blocking — only a horizontally huge slice does.) The final +`if (nblocks > max_sum_count)` guard (`:702`) is the hard ceiling — "over 17 billion points per PE" +for default settings, per the code comment (`niblocks*njblocks <= max_sum_count`, i.e. a slice with +more than `(2^17-2)^2 ~= 1.7e10` points). + +### 2.2 The kernel — quoted in full context (`:706-772`) + +```f90 +array_sum(:) = 0 + +do jb=1,njblocks ; do ib=1,niblocks + ! Use evenly distributed blocks, either floor(n / nblocks) or ceil(n / nblocks). + jbs = js + ((jb - 1) * nj) / njblocks + jbe = js + (jb * nj) / njblocks - 1 + ibs = is + ((ib - 1) * ni) / niblocks + ibe = is + (ib * ni) / niblocks - 1 + + block_sum(:) = 0 + block_max_pos = 0. ; block_max_neg = 0. + + ! Compute the sum of each block + do concurrent (j=jbs:jbe, i=ibs:ibe) & + DO_LOCALITY(local(r, e, rmag, lnan, lovf)) & + DO_LOCALITY(reduce(+: block_sum)) & + DO_LOCALITY(reduce(max: block_max_pos, block_max_neg, inan, iovf)) + + ! Convert array(i,j) to EFP form + r = descale * array(i,j) + call efp_decompose(r, e, rmag, lnan, lovf) + + inan = max(inan, lnan) + iovf = max(iovf, lovf) + + if (r >= 0.) then + if (rmag > block_max_pos) block_max_pos = rmag + else + if (rmag > block_max_neg) block_max_neg = rmag + endif + + ! Add the EFP result (including potential carry bits) + block_sum(:) = block_sum(:) + e(:) + enddo ; enddo + + array_sum(:) = array_sum(:) + block_sum(:) + + ! Redistribute carry bits across bins + ! For the final pass (or single pass) this is handled by ints_sum. + b = (jb - 1) * niblocks + ib + if (b < nblocks) call carry_overflow(array_sum, prec_error) + + max_pos = max(max_pos, block_max_pos) + max_neg = max(max_neg, block_max_neg) +enddo + +ints_sum(:) = ints_sum(:) + array_sum(:) +call carry_overflow(ints_sum, prec_error) +``` + +### 2.3 Why `reduce(+:block_sum)` over exact integers is bit-identical regardless of scheduling + +`block_sum` is an `integer(kind=int64), dimension(efp_digits)` accumulator, and `e(:)` (one element's +EFP decomposition) is likewise exact int64s. The `do concurrent ... DO_LOCALITY(reduce(+: block_sum))` +directive (`DO_LOCALITY(X)` expands to `X` when `HAVE_FC_DO_CONCURRENT_LOCAL` is defined, else to a +bare `;` no-op — `do_concurrent_compat.h:6-10`; `reduce`/`local` are **Fortran 2023 `do concurrent` +locality specifiers**, part of the base language, *not* OpenMP clauses) tells the compiler it may +split the +reduction across threads/teams/warps in any grouping and combine partial sums in any order — which is +exactly the freedom **integer addition** tolerates without changing the result, given the block was +sized so no intermediate partial sum can exceed the guard-bit budget (§2.1). Contrast this with the +`reduce(max: block_max_pos, block_max_neg, inan, iovf)` reduction on the same line: `max` is also +associative/commutative and exact for reals (no rounding in comparing magnitudes), so it is equally +safe to parallelize — it is used only for overflow/NaN bookkeeping and the diagnostic "largest term" +message, never for the sum itself. **No floating-point `+` reduction ever appears in this kernel** — +every quantity that GPU threads reduce into (`block_sum`, `max`/`min` trackers) is either an exact +integer or an exact-comparison real max, which is why arbitrary thread/team scheduling cannot perturb +a single bit of the final answer. + +The one place order still matters is the *serial* host loop over blocks (`do jb=1,njblocks; do +ib=1,niblocks`) and the `carry_overflow` calls between blocks (`:749`, `:760`) — but that loop always +executes in the same fixed order (row-major block index `b`) on every run, independent of PE count or +GPU scheduling, so it introduces no non-determinism; it exists purely to keep `array_sum`/`ints_sum` +from overflowing between blocks, not to control summation order for correctness. + +**Latent build caveat.** The `HAVE_FC_DO_CONCURRENT_LOCAL` feature test +(`ac/m4/mom6_fc_do_concurrent_local.m4:18`) probes only a `local(a,b)` specifier — it does **not** +compile-test the `reduce(...)` specifier this kernel actually relies on (the m4 comment at `:4-5` +notes `LOCAL_INIT`, `SHARED`, `DEFAULT(NONE)` are also untested). So on a hypothetical compiler that +accepts `local` but not `reduce`, the macro would be defined and the `reduce(+:block_sum)` / +`reduce(max:...)` clauses would be emitted and fail to compile. This is not a reproducibility bug, but +it is the load-bearing assumption behind the whole GPU path: the port presumes a compiler (nvfortran) +where `local` support implies `reduce` support. + +--- + +## 3. `efp_decompose`: `pure` + `declare target`, and the `fix/nan_repro_sum` lesson + +### 3.1 Why `pure` + `!$omp declare target`, and flags instead of module globals + +```f90 +!> Decompose one real into its 6 signed EFP bin contributions. NaNs and +!! overflows are reported by flags, rather than the module-level error +!! logicals, so that the routine is free of side effects. +pure subroutine efp_decompose(r, e, rmag, is_nan, is_ovf) + !$omp declare target + real, intent(in) :: r + integer(kind=int64), intent(out) :: e(efp_digits) + real, intent(out) :: rmag + integer, intent(out) :: is_nan + integer, intent(out) :: is_ovf + ... +end subroutine efp_decompose +``` +(`MOM_coms.F90:775-821`) + +The module also carries two **module-level** `logical` flags used elsewhere in the file: +`overflow_error` and `NaN_error` (`:71-74`). Fortran's `pure` attribute forbids a procedure from +modifying any entity outside its own dummy-argument list (no writes to module variables, no I/O, no +`stop`), and a `pure` procedure additionally cannot call an `impure` one. `efp_decompose` therefore +cannot set `NaN_error`/`overflow_error` directly — instead it returns `is_nan`/`is_ovf` as ordinary +`intent(out)` integer flags (`1` if a NaN/Inf or an unrepresentable magnitude was seen, else `0`), +which the caller (`increment_block_ints`) folds into thread-local `inan`/`iovf` accumulators via +`DO_LOCALITY(reduce(max: ..., inan, iovf))` (`:724`) and only *after* the parallel region converts them +to the module flags (`:769-771`): + +```f90 +if (inan /= 0) NaN_error = .true. +if (iovf /= 0) overflow_error = .true. +``` + +This buys two things simultaneously: (1) `efp_decompose` qualifies as `pure`, which is required for it +to be callable inside a `do concurrent` reduction region and legally `!$omp declare target`-able (a +device-resident routine must not perform host-only side effects like setting a host module variable); +and (2) the NaN/overflow signal is carried out of the parallel region via an *exact-max* reduction +(`inan`, `iovf` are `0`/`1` integers — associative, no rounding), so detecting a NaN anywhere in the +domain is itself scheduling-independent, consistent with the rest of the reproducibility design. + +### 3.2 What `fix/nan_repro_sum` changed, and why it couldn't stay `pure` + +Two commits on the (unmerged, stale) branch `fix/nan_repro_sum`, branched from `c82e1254a` — an +ancestor *older than* the `8593a732a` block-based rewrite, i.e. this branch still has the +three-way-dispatch pre-GPU-block version of the summation code, not the version described in §2: + +- `0ac71d482` "fix nan in repro sum for large domain sizes" — the pre-`8593a732a` code chose between + `increment_ints_2d` (small tile, on-device `do concurrent`), `increment_ints_faster` (medium tile, + scalar accumulate), and a fully scalar `increment_ints`+`real_to_ints` loop (large tile) based on + `(je+1-js)*(ie+1-is)` vs. `max_count_prec`. The large-tile branches called `increment_ints_faster`/ + `real_to_ints`, which had never been ported to run `!$omp declare target` and so implicitly read the + (device-resident) `array` through host memory — returning NaN whenever `array` lived only on the + GPU. The commit message: *"Larger domains triggered increment_ints_faster which was not ported. + Folded the routine into a single one and labelled it pure to reduce bloat."* The fix collapsed all + three paths into one `increment_ints_2d` that internally chunks the flattened `(i,j)` window into + `csize = max_count_prec - 1`-sized pieces, each summed by a `do concurrent` reduction into a fresh + chunk accumulator, carried, and folded into the running total — structurally the same "partition + into carry-safe chunks, do-concurrent-reduce each chunk over exact integers" idea as + `increment_block_ints` in §2, arrived at independently and by a different author. The routine was + marked `pure` in this commit. +- `939d06704` "cant be pure" — one commit later, `pure` was **removed** from both `increment_ints_2d` + and `carry_overflow`: + ```f90 + -pure subroutine increment_ints_2d(array, is, ie, js, je, descale, ints_sum, max_mag_term, prec_error) + +subroutine increment_ints_2d(array, is, ie, js, je, descale, ints_sum, max_mag_term, prec_error) + ... + -pure subroutine carry_overflow(int_sum, prec_error) + +subroutine carry_overflow(int_sum, prec_error) + ``` + **Why it couldn't stay `pure`:** `carry_overflow` sets the module-level `overflow_error = .true.` + when a carried sum exceeds `prec_error` (`MOM_coms.F90:841-843` in the current tree; same logic + existed on the branch) — a write to non-local (module) state, which the Fortran standard forbids + inside a `pure` procedure. `increment_ints_2d` calls `carry_overflow` once per chunk, so it too + cannot be `pure` (a `pure` procedure may only call other `pure` procedures). This is exactly the + discipline `efp_decompose` was designed around in §3.1 — return flags through the argument list + instead of writing a module global — but the `fix/nan_repro_sum` branch's `carry_overflow` was never + refactored that way, so the compiler (correctly) rejected `pure` on the caller chain. The lesson + generalizes: **a routine can only be `pure`/`declare target` if every side effect, including + warning/error flags, is threaded through `intent(out)` dummy arguments — not module variables** — + which is precisely what `efp_decompose`'s doc comment (`:776-777`, "reported by flags... so that the + routine is free of side effects") states as a design rule, and what this branch had to relearn the + hard way for `carry_overflow`. + + Because `fix/nan_repro_sum` predates the `8593a732a` rewrite, the block-based `increment_block_ints` + in the current `dev/gpu` tree independently avoids the same bug class: it never has a + size-dependent CPU/host fallback branch, so there is no code path in the current tree that silently + reads a device-resident array through host memory. The branch is best read as a documented case study + in the purity constraint, not as an outstanding patch that still needs to land. + +--- + +## 4. Where reproducing sums are invoked in the timestep + +Per `00-architecture.md` §4.2, the pure dycore compute kernels (`continuity_PPM`, `CorAdCalc`, +`PressureForce_FV`, `hor_visc`) contain **no reproducing sums** — communication and diagnostics are +hoisted out of the hot compute path into the driver and the diagnostics layer. Confirmed call sites: + +- **`src/diagnostics/MOM_sum_output.F90`, `write_energy`** — the periodic (not every-timestep) global + energy/mass/heat/salt diagnostic: + - `:566` — `mass_tot = reproducing_sum(tmp1, ..., sums=mass_lay, EFP_sum=mass_EFP, unscale=...)` + (total ocean mass + per-layer masses) + - `:576` — `vol_tot = reproducing_sum(tmp1, ..., sums=vol_lay, unscale=...)` (non-Boussinesq volume) + - `:746` — `PE_tot = reproducing_sum(PE_pt, ..., sums=PE, unscale=RZL4_T2_to_J)` (potential energy) + - `:758` — `KE_tot = reproducing_sum(tmp1, ..., sums=KE, unscale=RZL4_T2_to_J)` (kinetic energy) + - `:770-773` — `salt_EFP = reproducing_sum_EFP(Salt_int, ...)`, `heat_EFP = reproducing_sum_EFP(Temp_int, ...)` + (returned as `EFP_type` so they can be exactly accumulated across calls before conversion to real) + - `:776-781` — the salt/heat EFP values plus three running-total `CS%*_EFP` fields are packed into a + 5-element `EFP_type` array and reduced across PEs in one call, `EFP_sum_across_PEs(EFP_list, 5)` + (`:778`) — "Combining the sums avoids multiple blocking all-PE updates" (comment at `:775`). +- **`src/core/MOM_forcing_type.F90`, forcing/flux diagnostics** — every "total_*" (area-integrated) and + "*_ga" (area-averaged) diagnostic funnels through `MOM_spatial_means`, which itself calls + `reproducing_sum` once per invocation: + - `global_area_integral` calls sites at `:2848, 2875, 2908, 2919, 2933, 2945, 2957, 2969, 2981, 2989, + 2997, 3005, 3026, 3034, 3041, 3047, 3054, 3061, 3068, 3075, 3082` (net P-E, net mass in/out, evap, + precip, runoff, and every heat_content_* term) — each ultimately reaches + `reproducing_sum(tmpForSumming, unscale=temp_scale)` at `src/diagnostics/MOM_spatial_means.F90:234`. + - `global_area_mean` calls at `:2852, 2923, 2937, 2949, 2961, 2973` -> `reproducing_sum(...) * + G%IareaT_global` at `MOM_spatial_means.F90:84`. +- **`src/diagnostics/MOM_checksums.F90`** also calls `reproducing_sum` itself, for the `aMean` statistic + reported alongside every bitcount checksum (`subStats`, `MOM_checksums.F90:550`: + `aMean = reproducing_sum(array(HI%isc:HI%iec,HI%jsc:HI%jec))`). + +**Net picture:** reproducing sums appear only in (a) the low-frequency `write_energy` global-budget +diagnostic and (b) forcing/flux area-integral diagnostics, both of which run far less often than the +per-timestep dycore kernels and both of which are *diagnostic outputs*, not part of the prognostic +state update — consistent with §4.2 of `00-architecture.md` ("the pure compute kernels ... contain no +... reproducing sums by design"). + +--- + +## 5. `MOM_checksums.F90` — verifying a port bit-for-bit + +### 5.1 The bitcount checksum (`popcnt` mod 10^9) + +```f90 +integer, parameter :: bc_modulus = 1000000000 !< Modulus of checksum bitcount ! :112 + +!> Does a bitcount of a number by first casting to an integer and then using BTEST +!! to check bit by bit +integer function bitcount(x) + real, intent(in) :: x + integer, parameter :: xk = kind(x) + ! NOTE: Assumes that reals and integers of kind=xk are the same size + bitcount = popcnt(transfer(x, 1_xk)) +end function bitcount +``` +(`MOM_checksums.F90:2678-2687`) + +`popcnt` (population count, i.e. number of set bits) applied to the raw bit pattern of a `real` +(reinterpreted via `transfer` as an integer of the same storage size) is an **exact fingerprint of the +IEEE bit pattern** — it is not a numeric function of the value, it's a function of the bits. Any +change to even the last mantissa bit (a genuine bitwise-non-reproducibility bug — rounding difference, +reordered FP op, different fused-multiply-add contraction, etc.) changes `popcnt` and thus the +checksum; conversely, if two runs (e.g., CPU vs. GPU, or before/after a k-blocking refactor) print +identical checksums for every field, the fields are bit-identical, not merely "close." The per-element +`bitcount` results are summed with plain **exact integer addition** (`subchk = subchk + bc`, +`:527` in `chksum_h_2d`'s internal `subchk` function) and then reduced across PEs with +`sum_across_PEs(subchk)` (`:529`, also an exact integer reduction) before being folded into a fixed +range with `mod(subchk, bc_modulus)` (`:530`) purely so the printed number stays a manageable ~9-digit +value — the same exact-integer-reduction principle as the EFP reproducing sum (§1-2), applied here to +a diagnostic fingerprint instead of a physical total. This pattern (`subchk`/`bitcount`/`bc_modulus`) +recurs in every stagger-specific checksum routine: `chksum_h_2d` (`:389-557`), `chksum_B_2d` +(`:690-878`), `chksum_u_2d` (`:1007-1208`), `chksum_v_2d` (`:1211-1412`), and their 3-D and +pair-checksum (`chksum_pair_h_2d`, `chksum_uv_2d`, etc.) counterparts, exposed through the generic +interfaces `hchksum`/`Bchksum`/`uchksum`/`vchksum`/`qchksum`/`chksum` (`:22-23, 60-77`). Each of these +routines also calls `is_NaN` (generic over 0d/1d/2d/3d, `:85-87`) before checksumming, so a NaN +introduced by a bad port is caught with a `FATAL` error naming the field (`chksum_error(FATAL, 'NaN +detected: '//trim(mesg))`, e.g. `:435-436`) rather than silently corrupting the checksum. + +### 5.2 Device -> host checksum transfers (`b29b27150`) + +Checksum routines are host-only code (they call `MOM_error`, do character formatting, and write to +`error_unit`) and are never `!$omp declare target`. Any array that is device-resident (mapped via +`enter data`/`omp target` and updated only by device kernels) must be explicitly synced back to the +host with `!$omp target update from(...)` **immediately before** it is passed into a checksum call, or +the checksum silently reads stale/uninitialized host memory instead of the current device values — +which looks like a reproducibility failure but is actually a missing transfer. Commit `b29b27150` +("btstep: Update GPU checksum transfers", `src/core/MOM_barotropic.F90`, +8 lines) added exactly these +missing update directives ahead of debug checksums inside `btstep`: + +```f90 +!$omp target update from(CS%q_D) +call Bchksum(CS%q_D, "BT PV (q_D)", CS%debug_BT_HI, ...) +... +!$omp target update from(q) +call Bchksum(q, "BT PV (q)", CS%debug_BT_HI, ...) +!$omp target update from(DCor_u, DCor_v) +call uvchksum("BT DCor_[uv]", DCor_u, DCor_v, G%HI, ...) +!$omp target update from(Cor_ref_u, Cor_ref_v) +call uvchksum("BT Cor_ref_[uv]", Cor_ref_u, Cor_ref_v, CS%debug_BT_HI, ...) +!$omp target update from(uhbt0, vhbt0) +call uvchksum("BT [uv]hbt0", uhbt0, vhbt0, CS%debug_BT_HI, ...) +... +!$omp target update from(visc_rem_u, visc_rem_v) +call uvchksum("BT visc_rem_[uv]", visc_rem_u, visc_rem_v, G%HI, ...) +!$omp target update from(bc_accel_u, bc_accel_v) +call uvchksum("BT bc_accel_[uv]", bc_accel_u, bc_accel_v, G%HI, ...) +!$omp target update from(CS%IDatu, CS%IDatv) +call uvchksum("BT IDat[uv]", CS%IDatu, CS%IDatv, G%HI, ...) +``` + +**Porting rule:** every debug/diagnostic checksum call on a mapped array needs its own `target update +from` immediately upstream (an existing transfer for a *different* array, e.g. the pre-existing +`!$omp target update from(CS%frhatu, CS%frhatv)` a few lines above in the same routine, does **not** +cover a different array). This is a distinct failure mode from a genuine reproducibility bug and +should be the first thing checked when a checksum "mismatch" appears after porting a new kernel. + +### 5.3 Rotated-grid checksums + +`MOM_checksums` supports comparing a field computed on a quarter-turned ("rotated") test grid against +the same field on the canonical grid, a MOM6 technique for catching orientation-dependent bugs (stencil +asymmetries, sign errors in vector components, etc.) — a different axis of "reproducibility" from +GPU-vs-CPU but implemented with the same machinery. Each stagger-specific checksum routine takes the +input array on the *model's* (possibly rotated) index space and un-rotates it before checksumming, e.g. +`chksum_h_2d` (`MOM_checksums.F90:389-432`): + +```f90 +turns = HI_m%turns +if (modulo(turns, 4) /= 0) then + allocate(HI) + call rotate_hor_index(HI_m, -turns, HI) + allocate(array(HI%isd:HI%ied, HI%jsd:HI%jed)) + call rotate_array(array_m, -turns, array) +else + HI => HI_m + array => array_m +endif +``` + +`rotate_array`/`rotate_array_pair`/`rotate_vector` (imported from `MOM_array_transform`, +`MOM_checksums.F90:8-9`) implement the four index-map rotations (`+90`: transpose + row-reverse; +`180`: row+column reversal; `-90`: row-reverse + transpose, per the module doc comment in +`src/framework/MOM_array_transform.F90:6-13`) so that a field computed on a rotated test grid and +checksummed through `chksum_h_2d`/`chksum_B_2d`/`chksum_u_2d`/`chksum_v_2d` is transformed back onto +the canonical index space first — meaning the printed checksum for a "rotated" run and an "unrotated" +run of a correctly-ported, order-preserving kernel should be bit-identical, and any mismatch flags +either a genuine order-of-operations bug or a stencil that implicitly assumes a fixed index direction +(e.g. hard-coded `i+1` where a rotation-safe kernel should use a metric-relative offset). + +--- + +## 6. Order-of-operations rules for a porter (prescriptive) + +These are the rules an agent must apply, in order, when porting any module that touches a sum, +average, reduction, or restart-critical total. Each is a decision an agent can execute mechanically; +the grounding for each is cited so it can be re-verified. + +### 6.1 Deciding whether a loop is safe to parallelize + +1. **Classify the reduction operator before touching the loop.** + - **`+` over reals → NOT safe. STOP.** A `do concurrent`/`omp target` loop that sums or averages + real values changes FP operation order the instant it is parallelized (thread/team/warp order is + not sequential loop order), and IEEE `+` is non-associative. A raw `reduce(+: real_var)` in a + device loop is a reproducibility bug even if the CPU serial version is "correct". + - **`max`/`min`/`.and.`/`.or.` over reals or integers → SAFE.** Comparison and logical fold are + exact (no rounding) and associative, so any grouping gives the same result. The kernel's + `reduce(max: block_max_pos, block_max_neg, inan, iovf)` (`:724`) is the canonical safe use — it + carries the largest-magnitude term and the NaN/overflow flags out of the parallel region without + threatening reproducibility (§2.3, §3.1). + - **`+` over exact `int64` (or `int64` arrays) → SAFE, *if* the block is carry-bounded.** This is + the whole EFP trick: `reduce(+: block_sum)` over `integer(int64)` digits is order-invariant + because integer `+` is associative up to overflow, and the block was sized so no partial sum + overflows (§2.1). `MOM_checksums`'s `subchk = subchk + bc` over `bitcount` integers (`:527`) is + the same pattern for a fingerprint. + +2. **If you need a reproducible real sum, route it through the EFP path — never hand-roll.** Use + `reproducing_sum` / `reproducing_sum_EFP` (public interfaces, `MOM_coms.F90:80-90`). Return an + `EFP_type` (via `reproducing_sum_EFP` or the `EFP_sum=` argument) when the total must be + accumulated across multiple calls or PEs *before* conversion to real — converting to real early + and re-summing reintroduces FP non-associativity. `write_energy` packs 5 running EFP totals into + one `EFP_sum_across_PEs(EFP_list, 5)` for exactly this reason (§4, `MOM_sum_output.F90:775-778`). + +3. **When a slice is "too large" for one reduction pass, partition into carry-safe blocks and reduce + each block exactly** (§2.1's `max_sum_count = max_summands - 2`, `niblocks`/`isize_max`/`jsize`/ + `njblocks`). Do **not** add a size-dependent scalar/host fallback branch: that pattern is exactly + what produced the NaN on `fix/nan_repro_sum` (§3.2), where the large-domain branch called an + unported routine that read a device-resident array through host memory. In the current tree + `increment_block_ints` has a single code path for all sizes, by design. + +### 6.2 Restructuring without perturbing arithmetic + +4. **k-blocking / tiling may change loop *structure* only, never the arithmetic order within a + column or stencil.** Rewrites must move already-computed scalar operations around; they must not + re-associate a running accumulation (`00-architecture.md` §5). `increment_block_ints` embodies + this: it partitions into whole rectangular blocks summed with a fixed, deterministic per-block + carry step (`carry_overflow` between blocks, `:749`, `:760`) — not an arbitrarily scheduled global + reduction. The serial host block-loop runs in fixed row-major order every time (§2.3). + +5. **Extract shared math into `pure` (host) or `pure` + `!$omp declare target` (device) helpers** + so host and device evaluate bit-identically from one source (`efp_decompose`, §3.1). This is the + preferred restructuring tool (`00-architecture.md` §0.2). + +6. **A `pure`/`declare target` routine may not write module or host-global state — thread every + error/warning/overflow flag out through `intent(out)` dummy arguments.** `efp_decompose` returns + `is_nan`/`is_ovf` integer flags instead of setting the module `NaN_error`/`overflow_error` globals; + the caller folds them in *after* the parallel region (`:770-771`). The cautionary counter-example + is `fix/nan_repro_sum`'s `939d06704` ("cant be pure"): `carry_overflow` sets `overflow_error` + (`:841-843`), so neither it nor any caller that invokes it can be `pure` (§3.2). + +### 6.3 Compiler / language hazards that silently break bit-identity + +7. **Preserve parentheses; never let a rewrite re-associate an expression.** The Fortran standard + *forbids* a processor from breaking parenthesized sub-expressions (`(a+b)+c` must not become + `a+(b+c)`), so parentheses are the porter's tool for pinning evaluation order in the compute + kernels themselves — do not "simplify" `(a*b) + (c*d)` into a rearranged form when refactoring, and + do not distribute/factor terms in a reconstruction loop. The EFP reconstruction `r = r + pr(i)*ints(i)` + (`ints_to_real`, `:578`) is a fixed 6-term unrolled loop precisely so its order is invariant. + +8. **FMA contraction is a bitwise hazard across CPU vs. GPU.** Fusing `a*b + c` into a single + fused-multiply-add changes the rounding (one rounding instead of two), so a kernel that contracts + on GPU but not on CPU (or vice-versa) yields a different last mantissa bit and a `MOM_checksums` + mismatch that is *not* an algorithm bug. The CPU reference build enables the FMA instruction + (`FCFLAGS_OPT = -g -O3 -mavx -mfma`, `.testing/README.rst:147`); no explicit `-ffp-contract` / + `-Mnofma` pin was found anywhere in `ac/`, the test `Makefile`s, or `.testing/`. + > **Resolved (2026-07-14):** Contraction *is* pinned in the canonical NVHPC toolchain: + > `mkmf/templates/ncrc5-nvhpc.mk` (and `ncrc-nvhpc.mk`) put `-Mnofma` — plus `-Mdaz` — in the + > **base** `FFLAGS`, for all build modes. So bit-identity does not rest on nvfortran and the CPU + > reference compiler happening to contract identically. The one action item that remains: the site + > GPU build harness is external to this repo, so confirm it inherits `-Mnofma`. If it does, this + > hazard is closed; if it does not, the fragility above is live. + +9. **The GPU reduction path assumes a compiler where `do concurrent local` implies `reduce` + support.** The `HAVE_FC_DO_CONCURRENT_LOCAL` autoconf probe tests only `local(a,b)`, not the + `reduce(...)` specifier the repro kernel emits (§2.3). This is a build-portability, not a + reproducibility, constraint — but a porter adding a new `DO_LOCALITY(reduce(...))` kernel inherits + the same assumption. + +### 6.4 Verification (always, non-negotiable) + +10. **Before any checksum on a device-resident array, emit `!$omp target update from(...)` for that + exact array.** A missing transfer looks identical to a reproducibility failure but is really a + stale-host-memory read; an existing transfer for a *different* array does not cover it (§5.2, + commit `b29b27150`). Check this *first* when a checksum "mismatch" appears after a new port. + +11. **Verify every port with `MOM_checksums`** (`hchksum`/`Bchksum`/`uchksum`/`vchksum`), comparing + CPU vs. GPU and, where the config supports it, rotated vs. unrotated runs (§5.3). The `popcnt` + bitcount (mod `bc_modulus = 10^9`, §5.1) is an exact bit-pattern fingerprint: it passes only on + bit-identical fields, never on "acceptably close" ones. A rotated-vs-unrotated mismatch + additionally flags a stencil that hard-codes an index direction (e.g. literal `i+1`) instead of a + metric-relative offset. + +--- + +## References + +- `src/framework/MOM_coms.F90` — EFP type/params (`:31-113`), `reproducing_EFP_sum_2d` (`:121-232`), + `reproducing_sum_2d` (`:239-347`), `reproducing_sum_3d` (`:353-523`), `real_to_ints`/`ints_to_real` + (`:526-579`), `increment_ints`/`increment_block_ints` (`:583-772`), `efp_decompose` (`:778-821`), + `carry_overflow` (`:825-845`), `regularize_ints` (`:849-887`), `EFP_to_real` (`:937-943`). +- `src/framework/do_concurrent_compat.h` — `DO_LOCALITY(X)` macro (expands to `X` if + `HAVE_FC_DO_CONCURRENT_LOCAL`, else a no-op semicolon). +- `src/diagnostics/MOM_sum_output.F90` — `write_energy` reproducing-sum call sites (`:566, 576, 746, + 758, 770-781`). +- `src/diagnostics/MOM_spatial_means.F90` — `global_area_mean` (`:40-86`), `global_area_mean_v/_u` + (`:89-160`), `global_area_integral` (`:166-236`); all wrap `reproducing_sum`. +- `src/core/MOM_forcing_type.F90` — forcing/flux diagnostic call sites (`:2848-3082` and beyond) into + `global_area_integral`/`global_area_mean`. +- `src/framework/MOM_checksums.F90` — module header/interfaces (`:1-121`), `bc_modulus` (`:112`), + `chksum_h_2d`/`subchk`/`subStats` (`:389-557`), `bitcount` (`:2678-2687`). +- `src/framework/MOM_array_transform.F90` — `rotate_array`/`rotate_array_pair`/`rotate_vector` module + header describing the four rotation cases (`:1-60`). +- `config_src/drivers/unit_tests/test_reproducing_sum.F90` — reference unit test: standard vs. + reproducing vs. fast-reproducing sum agreement, exact analytic sum check, and order-invariance under + random element swaps (`randomly_swap_elements`, whole file). +- Commits: `8593a732a` (block-based GPU repro-sum rewrite), `be42560d9` and `813dc1f5b` (earlier + reproducing-sum/`write_energy` porting steps that `8593a732a` supersedes), `b29b27150` (btstep GPU + checksum transfer fixes). Branch `fix/nan_repro_sum` (`0ac71d482`, `939d06704`) — unmerged, based on + an ancestor (`c82e1254a`) that predates `8593a732a`; documents the NaN/"cant be pure" lesson rather + than an outstanding patch. + +--- + +## Verification notes + +Verified by an Opus verification agent against `dev/gpu` source + git only (no build/run), 2026-07. + +**Confirmed (checked line-by-line against source/git):** +- All EFP parameters and line numbers: `accum_width = digits(1_int64)` (=63) `:31`, `prec_width = 46` + `:33`, `guard_width = 17` `:35`, `max_summands = 2**17 = 131072` `:42`, `prec = 2**46` `:46`, + `efp_digits = 6` `:54`, `EFP_type` `:103`. Comment `:38-40` quoted correctly. +- **Carry-budget arithmetic re-derived independently and confirmed:** each EFP digit `e(n>=1)` holds + `< prec = 2^46`; summing `N` of them keeps `|block_sum| < N*2^46`, which stays inside the signed + `int64` range (`< 2^63`) iff `N <= 2^17 = max_summands`. `max_sum_count = max_summands - 2` `:671` + correctly reserves headroom for the two cumulant adds (`block_sum→array_sum`, `array_sum→ints_sum`). + The `nblocks > max_sum_count` FATAL guard `:702` and the "over 17 billion points per PE" + (`≈ (2^17-2)^2`) claim check out. +- `increment_block_ints` kernel `:706-772`, `efp_decompose` `pure`+`declare target` `:778-821`, + `carry_overflow` `:825-845` (sets module `overflow_error` `:841-843`), flag-folding `:770-771`, + `DO_LOCALITY` macro `do_concurrent_compat.h:6-10`, `sum_across_PEs(ints_sum, efp_digits)` `:226`. +- Git: `8593a732a` (M. Ward, "MOM_coms: GPU port of block-based repro sum"); `b29b27150` (M. Ward, + "btstep: Update GPU checksum transfers", **+8 lines**, all quoted directives verified against the + diff, incl. the pre-existing `frhatu/frhatv` transfer that does *not* cover neighbors); + `fix/nan_repro_sum` = `939d06704` "cant be pure" + `0ac71d482` (author **Jorge Galvez Vallejo**, not + M. Ward — supports the doc's "different author" claim), parented on `c82e1254a`, confirmed a genuine + ancestor of `8593a732a`. `0ac71d482`'s message quoted accurately. +- `MOM_checksums`: `bc_modulus = 10^9` `:112`, `bitcount = popcnt(transfer(...))` `:2680-2687`, + `subchk`/`sum_across_PEs`/`mod` `:527/529/530`, `subStats` `aMean = reproducing_sum(...)` `:550`, + rotate block `:422-432`; `MOM_array_transform.F90:8-13` rotation cases; call sites in + `MOM_sum_output.F90` (`:566,576,746,758,770,772,778`) and `MOM_spatial_means.F90` (`:84,234`) all + confirmed at the stated lines. + +**Corrected:** +1. §1 — `EFP_to_real` was described as "a thin wrapper (`ints_to_real`)"; it actually calls + `regularize_ints` *first* (`:941`), which is why `EFP1` is `intent(inout)`. Clarified. +2. §2.1 — **Substantive:** the claim that 3-D calls engage blocking via "many k-layers accumulating + into the same `ints_sum`" is wrong. `reproducing_sum_3d` sums each layer into a *separate* + `ints_sums(:,k)` accumulator (`:386,432,434-435`); k never enters the carry budget. Each + `increment_block_ints` call sums exactly one `ni×nj` 2-D slice — only horizontal extent can force + blocking. Rewritten. +3. §2.3 — `reduce`/`local` were called "OpenMP-style"; they are **Fortran 2023 `do concurrent` + locality specifiers** (base language). Corrected, and added the feature-detection caveat: the + autoconf probe (`mom6_fc_do_concurrent_local.m4:18`) tests only `local(a,b)`, not `reduce(...)`. +4. §1 — `pr` array was written with `prec`; it uses the *real* `r_prec` (`:56-57`). Fixed. Reference + line for `regularize_ints` tightened to `:849-887`. + +**Enhanced:** +- §6 restructured into prescriptive, agent-executable rules (6.1 classify-the-operator decision; + 6.2 restructuring; 6.3 compiler/language hazards; 6.4 verification). Added: **FMA-contraction** + hazard (grounded in `-mfma` at `.testing/README.rst:147`; no `-ffp-contract`/`-Mnofma` pin found in + `ac/`/`.testing/`), **parentheses-preservation** as a Fortran-standard anti-reassociation tool, and + the `reduce`-clause build-portability assumption. + +**Confidence.** High on all EFP/checksum mechanics, the carry math, and git provenance (directly +verified). The FMA hazard is a real and correctly-described class of bug, but it is not currently live: +contraction is pinned by `-Mnofma` in the base `FFLAGS` of the NVHPC mkmf templates (§6.3), so it bites +only if the site GPU build harness fails to inherit that flag. diff --git a/knowledge/gpu-knowledge/08-cross-module-inlining.md b/knowledge/gpu-knowledge/08-cross-module-inlining.md new file mode 100644 index 0000000..3494cfc --- /dev/null +++ b/knowledge/gpu-knowledge/08-cross-module-inlining.md @@ -0,0 +1,519 @@ +# Cross-module calls and inlining on `dev/gpu` + +> Drills into architecture doc §0.4, §5, §7.1, §7.5. Scope: every place a helper subroutine/function +> is called from inside a device compute region (`!$omp target`/`!$omp loop`/`do concurrent`), the +> directives and refactors that make that legal on nvfortran, the exact `-Minline` requirement, and +> the cases where duplication (not sharing) was used to route around the problem. + +--- + +## 1. The core rule + +**A subroutine/function called from inside an OpenMP `target` compute region must either (a) carry +`!$omp declare target` so the device compiler emits device code for it, or (b) be force-inlined at +the call site.** Getting this wrong does not reliably fail to compile — commit `3cb184edd` is +explicit that skipping it produces **silently wrong numerical results**, not a compile error or +crash. This is the single most expensive class of bug in the port because it doesn't show up until a +checksum diverges. + +> **Force-inline directive changed at HEAD — read this before trusting §2/§3 below.** The inline +> mechanism went through *three* forms, and the current tree (`dev/gpu` HEAD, after the merged +> k-blocking commit `93dbbd36e`) is on the **third**: +> 1. `3cb184edd` — build flag only: `-Minline=name:ratio_max,name:flux_elem` (MANDATORY). +> 2. `4e3f1b758` — source pragma `!NVF$ INLINE` above each of `flux_elem`/`flux_elem_OBC`/`ratio_max` +> (compile with `-Minline=pragma`); also dropped `thread_limit(128)`. +> 3. `93dbbd36e` — replaced `!NVF$ INLINE` with the Intel-style **`!DIR$ ATTRIBUTES FORCEINLINE :: `** +> on `flux_elem` (`MOM_continuity_PPM.F90:1086`) and `flux_elem_OBC` (`:1149`), and **removed the +> directive from `ratio_max` entirely** — its commit body: *"remove nvf inline and replace with intel +> forceinline / Significantly improves performance of blocked zonal/meridional_mass_flux at -O2."* +> +> So **there are currently zero `!NVF$ INLINE` directives in `src/`** (`grep -rF 'NVF$ INLINE' src/` +> returns nothing) and `ratio_max` now carries **no** inline directive at all despite still being +> called from `!$omp target`/`!$omp loop` regions (see §3). The prose below that says +> "`!NVF$ INLINE`" describes the historical `4e3f1b758` state; the catalogue rows have been corrected +> to HEAD. + +A second, distinct rule governs `class(*)`/polymorphic dispatch: passing a polymorphic `this` into a +device region causes **runtime errors** on nvfortran (§4), not silent wrongness — so the two failure +modes must be told apart when debugging. + +`do concurrent` regions are more forgiving: nvfortran's `-stdpar=gpu` lowering appears to handle +plain `elemental`/`pure` procedure calls without a mandatory `declare target` — only the definition +matters if it's *also* reached from an `!$omp target` region. The declare-target/inline requirement +bites specifically on the `!$omp target teams` / `!$omp target ... !$omp loop` idiom used for the +hand-tuned team-count kernels (§5 of `00-architecture.md`). Note that in the current tree the +`flux_elem`/`ratio_max` call sites in continuity are all inside that `!$omp target teams` + `!$omp +loop collapse(2)` idiom (e.g. `flux_elem` at `MOM_continuity_PPM.F90:715-724`, `ratio_max` at +`:768-769`), **not** a bare `do concurrent` — which is exactly why they need the FORCEINLINE +treatment (the earlier draft's "`ratio_max` inside a bare `do concurrent` at `:1622-1630`" example was +inaccurate: `:1622-1630` are `flux_elem` calls inside an `!$omp loop`). + +--- + +## 2. Catalogue of device-callable helpers + +All 21 `!$omp declare target` occurrences in `src/` (this total *includes* the one `!$omp declare +target(pr, I_pr)` for module-level constant arrays — it is one of the 21, not extra), plus the +force-inlined helpers in continuity. **At HEAD the inline directive is `!DIR$ ATTRIBUTES FORCEINLINE` +on `flux_elem`/`flux_elem_OBC` and none on `ratio_max`** (see the banner in §1); the rows below have +been corrected accordingly. "Caller directive" is the enclosing compute-region form that requires the +callee to be device-resident. + +| Helper | file:line (def) | Directive (at HEAD) | Caller / enclosing region | Reason | +|---|---|---|---|---| +| `flux_elem` (elemental subroutine) | `src/core/MOM_continuity_PPM.F90:1087` | `!DIR$ ATTRIBUTES FORCEINLINE :: flux_elem` (`:1086`) | `!$omp target teams` / `!$omp loop` in `zonal_mass_flux`/`meridional_mass_flux` (calls at `:719,1060,1457,1622,1626,1630,1823,2162,2464,2629,2632,2635`; note `:724,1065,1462,1828,2167,2469` are `flux_elem_OBC`) | Mandatory inline — commit `3cb184edd`: "Otherwise results are incorrect." Was `!NVF$ INLINE` (`4e3f1b758`); changed to Intel FORCEINLINE by `93dbbd36e`. | +| `flux_elem_OBC` (elemental subroutine) | `:1150` | `!DIR$ ATTRIBUTES FORCEINLINE :: flux_elem_OBC` (`:1149`) | OBC call sites listed above | Same as `flux_elem`; was also listed explicitly in the `-Minline=name:...` flag | +| `ratio_max` (pure function) | `:3086` | **none at HEAD** (was `!NVF$ INLINE` at `:3085`, removed by `93dbbd36e`) | `!$omp target`/`!$omp loop` in `zonal_mass_flux` (`:768,769,792,793,813,814,833,834,851,852`) and `meridional_mass_flux` (`:1872,1873,1897,1898,1917,1918,1938,1939,1955,1956`) | Originally mandatory-inline (`3cb184edd`); directive since removed — see §3 | +| `efp_decompose` (pure subroutine) | `src/framework/MOM_coms.F90:778` (directive `:779`) | `!$omp declare target` | `do concurrent` reduction loop in `increment_block_ints` (`:721-728`) | Per-point EFP bin decomposition inside the reproducing-sum block reduction; must be device-resident since the reduction body is compiled for target | +| module constants `pr`, `I_pr` | `src/framework/MOM_coms.F90:69` | `!$omp declare target(pr, I_pr)` | referenced from inside `efp_decompose`/reduction body | Data (not code), but same device-residency requirement extends to module-level constant arrays consumed by device code | +| `cuberoot` (elemental function) | `src/framework/MOM_intrinsic_functions.F90:51` (directive), `:50` (def) | `!$omp declare target` | called from `MOM_barotropic.F90`, EOS files, etc. inside `do concurrent`/`omp target` | Replaces `x**(1/3)` (transcendental `exp/log` lowering) with a deterministic bit-exact iterative kernel; must be device-resident everywhere it's used | +| `nth_root` (elemental function) | `:133` (directive), `:132` (def) | `!$omp declare target` | `MOM_barotropic.F90`'s `bt_rem = av_rem**Instep` | Replaces `x**(1/n)` for general integer `n`; same transcendental-lowering problem as `cuberoot` | +| `rescale_cbrt` (pure subroutine) | `:181` (directive), `:180` (def) | `!$omp declare target` | called from `cuberoot` | Helper to `cuberoot`; must be device-resident since its caller is | +| `descale` (pure function) | `:246` (directive), `:245` (def) | `!$omp declare target` | called from `cuberoot` | Same as `rescale_cbrt` | +| `find_coupling_coef_gl90` | `src/parameterizations/vertical/MOM_vert_friction.F90:437` (directive), `:436` (def) | `!$omp declare target` | `!$omp target teams distribute parallel do collapse(2)` in `vertvisc_coef` (called `:1600,1904`) | Per-column GL90 vertical-viscosity coupling coefficient, called once per (I,j) inside the teams-distribute loop at `:1443` | +| `find_coupling_coef_k` (pure subroutine) | `:2101` (directive), `:2099` (def) | `!$omp declare target` | same teams-distribute loop, called `:1594,1898` | BBL-aware coupling coefficient | +| `find_coupling_coef` (non-pure subroutine) | `:2611` (directive), `:2609` (def) | `!$omp declare target` | same loop, called `:1679,1983` (ice-shelf branch) | Shelf-drag coupling coefficient variant | +| `find_L_open_uniform_slope` (pure subroutine) | `src/parameterizations/vertical/MOM_set_viscosity.F90:1251,1258` (directive, ×2) `:1241` (def) | `!$omp declare target` | called from BBL open-area column kernels inside device loops in `set_viscous_ML`/related | Column-local open-fraction geometry, one of 4 `find_L_open_*` variants selected per bottom-shape case | +| `find_L_open_concave_trigonometric` | `:1294,1316` (dir, ×2), `:1283` (def) | `!$omp declare target` | same family | ditto (uses `atan`-derived constant `C2pi_3`, still device-safe since it's a compile-time parameter) | +| `find_L_open_concave_iterative` | `:1389,1438` (dir, ×2), `:1378` (def) | `!$omp declare target` | same family | Newton-iteration fallback when the closed-form concave case is ill-conditioned | +| `test_L_open_concave` | `:1718,1742` (dir, ×2), `:1705` (def) | `!$omp declare target` | same family (diagnostic/verification path) | Consistency check on `find_L_open_concave_*` outputs, itself device-resident | +| `find_L_open_convex` | `:1802,1836` (dir, ×2), `:1788` (def) | `!$omp declare target` | same family | Convex bottom-shape case; iterates with `maxitt` | +| `set_v_at_u` (pure function) | `:1966` (dir, ×1), `:1951` (def) | `!$omp declare target` | thickness-weighted interpolation used inside device column loops | Cross-staggering interpolation (v at u-points) | +| `set_u_at_v` (pure function) | `:2012` (dir, ×1), `:1997` (def) | `!$omp declare target` | ditto | u at v-points | + +**Note on the double directive:** in `MOM_set_viscosity.F90`, each of the 5 `find_L_open_*`/ +`test_L_open_concave` routines carries `!$omp declare target` **twice** — once immediately after the +dummy-argument declarations, once again after the local-variable declarations, both before any +executable statement (e.g. lines 1251 and 1258 both sit inside `find_L_open_uniform_slope`, which +starts at 1241 and has no other procedure boundary between them). `set_v_at_u`/`set_u_at_v` carry it +only once. This accounts for 12 total `declare target` occurrences across 7 subroutines/functions in +this file (5×2 + 2×1 = 12) and is presumably harmless (a repeated directive on the same scope), but is +catalogued here since it's an idiosyncrasy specific to this file — the pattern is not used anywhere +else in the 21-occurrence inventory. + +Total: 21 `!$omp declare target` directives (matches the count in `00-architecture.md` §6.1) across +4 files (`MOM_coms.F90` (2: module constants `declare target(pr, I_pr)` + `efp_decompose`), +`MOM_intrinsic_functions.F90` (4), `MOM_vert_friction.F90` (3), `MOM_set_viscosity.F90` (12, including +the double-directive idiosyncrasy noted below)) — verified by `grep -rn "declare target" src/ | wc -l` += 21. **Force-inline in `MOM_continuity_PPM.F90` at HEAD: 2 `!DIR$ ATTRIBUTES FORCEINLINE` directives +(`flux_elem` `:1086`, `flux_elem_OBC` `:1149`), and zero on `ratio_max`.** (The "3 `!NVF$ INLINE` +sites" phrasing in `00-architecture.md` §7.5 and §6.1 predates `93dbbd36e` and is now stale — there +are 0 `!NVF$ INLINE` directives in the tree.) + +**Note on "cross-module":** in the strict sense (helper defined in a *different module* than the +caller) only `cuberoot`/`nth_root` (`MOM_intrinsic_functions` called from `MOM_barotropic`/EOS) and +the EOS `_loc` functions (defined in `MOM_EOS_Wright`/`MOM_EOS_Roquet_rho`, dispatched from +`MOM_EOS`) qualify. `flux_elem`/`ratio_max`, the `find_coupling_coef*` family, and the `find_L_open_*` +family are all same-module (private module procedures called from a sibling subroutine in the same +file) — but they hit the *identical* compiler problem, because the OpenMP device-code generation +model treats "outside the lexical scope of the enclosing `!$omp target` construct" as the boundary +that requires `declare target`, not the Fortran module boundary. The knowledge-base title tracks the +pain point ("procedure call inside a device loop"), which is broader than literal cross-`module` +calls. + +--- + +## 3. The exact `-Minline` requirement + +Commit `3cb184edd` ("use openmp instead of openacc", `src/core/MOM_continuity_PPM.F90`, +169/−198) +is the OpenACC→OpenMP translation of `MOM_continuity_PPM.F90` and states the rule verbatim: + +> IMPORTANT: However for OpenMP, inlining of ratio_max and flux_elem is MANDATORY. do so with +> `-Minline=name:ratio_max,name:flux_elem`. Otherwise results are incorrect. + +That commit's translation table (from the same message) is itself useful background for the OpenACC→ +OpenMP mapping used throughout the port: + +> outer parallel region followed by multiple inner acc loops is equivalent to an outer omp target +> followed by multiple inner omp loops. oacc parallel loop seems to be equivalent to omp target loop. + +Follow-up commit `4e3f1b758` ("add !NVF\$ INLINE to ratio_max flux_elem") replaces the build-flag +requirement with a source-level directive, so the build system no longer has to enumerate function +names: + +> instead of compiling with `-Minline=name:flux_elem,name:flux_elem_OBC,name:ratio_max`, can compile +> with `-Minline=pragma` instead. + +Diff (`4e3f1b758`) adds `!NVF$ INLINE` directly above each of the three definitions: +```fortran +!> Evaluates the zonal mass or volume fluxes in an element. +!NVF$ INLINE +elemental subroutine flux_elem(u, h, h_p1, h_L, h_L_p1, h_R, h_R_p1, uh, duhdu, visc_rem, & + G_dy_Cu, G_IareaT, G_IareaT_p1, G_IdxT, G_IdxT_p1, dt, & + vol_CFL, por_face_area) +``` +```fortran +!> Return the maximum ratio of a/b or maxrat. +!NVF$ INLINE +pure function ratio_max(a, b, maxrat) result(ratio) +``` +The same commit also **removes** a `thread_limit(128)` clause from the `!$omp target teams +num_teams(nteams)` construct that wraps these calls (kept as `num_teams(nteams)` only, verified at +`MOM_continuity_PPM.F90:707,1811`) — inlining changed the register/resource footprint enough that the +previous thread-limit tuning was dropped. + +**Third form (current HEAD) — commit `93dbbd36e`.** The k-blocking-continuity merge ("Use blocking in +k dimension for continuity reconstruction (#165)") replaced the `!NVF$ INLINE` pragmas with the +Intel-style `!DIR$ ATTRIBUTES FORCEINLINE :: ` directive on `flux_elem` and `flux_elem_OBC`, +and **deleted the directive from `ratio_max` outright.** Its commit body states the motivation: +*"remove nvf inline and replace with intel forceinline / Significantly improves performance of blocked +zonal/meridional_mass_flux at -O2."* Diff evidence (`git show 93dbbd36e -- src/core/MOM_continuity_PPM.F90`): +```diff +-!NVF$ INLINE ++!DIR$ ATTRIBUTES FORCEINLINE :: flux_elem + elemental subroutine flux_elem(...) +-!NVF$ INLINE ++!DIR$ ATTRIBUTES FORCEINLINE :: flux_elem_OBC + elemental subroutine flux_elem_OBC(...) +-!NVF$ INLINE + pure function ratio_max(a, b, maxrat) result(ratio) +``` +So at HEAD the `-Minline=pragma` recipe no longer targets `!NVF$ INLINE` (there are none); inlining is +now driven by the `!DIR$ ATTRIBUTES FORCEINLINE` directive (which nvfortran honours) on the two +`flux_elem` routines. + +> **Resolved (2026-07-14):** The gap is a deliberate removal resting on implicit device codegen. +> `93dbbd36e` removed `!NVF$ INLINE` from `ratio_max` without replacement while giving +> `flux_elem`/`flux_elem_OBC` FORCEINLINE, and **no `-Minline` exists in any in-repo or +> mkmf-template build config** — so no flag is quietly standing in for the directive. `ratio_max` is +> still called from `!$omp target`/`loop` regions in `MOM_continuity_PPM.F90`. Correctness at HEAD +> therefore rests on nvfortran implicitly compiling/inlining a small same-file `pure` function for +> the device: empirically fine on the tested toolchain (the commit is merged and checksum-gated), but +> fragile. **Recommendation:** add `!DIR$ ATTRIBUTES FORCEINLINE :: ratio_max` for parity with +> `flux_elem`. Never imitate the gap in new code. + +**Without either the flag or the pragma:** per `3cb184edd`'s commit message, the OpenMP target +version of `zonal_mass_flux`/`meridional_mass_flux` produces **incorrect results** (not a build +failure) if `flux_elem`/`ratio_max` are compiled as genuine out-of-line device subroutine calls +instead of being inlined. The exact failure mode (register spill, missing device symbol resolution at +link time, or a silent no-op) is not spelled out in the commit — only the empirical fact that results +are wrong — so this is catalogued as an observed nvfortran limitation, not a fully diagnosed root +cause. + +--- + +## 4. The pure/elemental refactoring pattern (the blessed fix) + +The house style for "this needs restructuring for the device, but must not change one bit of +arithmetic" is: **extract the innermost, side-effect-free arithmetic into a `pure`/`elemental` +procedure, and never touch the expressions themselves.** Four concrete instances: + +### 4.1 `efp_decompose` — reproducing-sum decomposition (`MOM_coms.F90:779`) + +```fortran +!> Decompose one real into its 6 signed EFP bin contributions. NaNs and +!! overflows are reported by flags, rather than the module-level error +!! logicals, so that the routine is free of side effects. +pure subroutine efp_decompose(r, e, rmag, is_nan, is_ovf) + !$omp declare target +``` +The doc comment is explicit about *why* it's `pure`: side-effect-freedom (no touching the module-level +`overflow_error`/`NaN_error` flags) is what makes it safe to call from inside a `do concurrent` +reduction body (`increment_block_ints`, `:721-728`) without breaking the compiler's ability to reason +about the loop. Errors are returned as `is_nan`/`is_ovf` output arguments and only folded into the +module flags by the (non-pure, host-side) caller after the reduction completes. + +### 4.2 EOS `_loc` free functions — the `this`-copy fix (§7.1 of `00-architecture.md`) + +Origin commit `52a1b3954` ("Added local versions of density_elem and density_derivs without "this" +argument", `MOM_EOS_Wright.F90`) states the problem directly in a doc comment it introduces: + +```fortran +!> Wrapper for density_elem_buggy_Wright_loc created to preserve API while calling +!! density_elem_buggy_Wright without "this" variable that causes runtime errors on +!! gpu runs with nvfortran. +real elemental function density_elem_buggy_Wright(this, T, S, pressure) + class(buggy_Wright_EOS), intent(in) :: this !< This EOS + ... + density_elem_buggy_Wright = density_elem_buggy_Wright_loc(T, S, pressure) +end function density_elem_buggy_Wright +``` +The original type-bound elemental (`class(this)` dummy) is kept **only as a thin wrapper** for API +compatibility with code that still dispatches through it; the arithmetic itself was moved verbatim +into a new free function with the identical body and an `_loc` suffix, dropping the `this` argument +entirely. Later commit `7c7af5572` ("EOS: 2D and 3D density implementations of methods (#185)") +extends the same split to `calculate_density_derivs_3d`/`calculate_stanley_density_2d`/ +`calculate_density_second_derivs_2d` in both `MOM_EOS_Wright.F90` and `MOM_EOS_Roquet_rho.F90`. + +The `_loc` functions are then called from `do concurrent (k,j,i)` device loops directly, e.g. +`MOM_EOS_Wright.F90:1058`: +```fortran +do concurrent (k=ks:ke, j=js:je, i=is:ie) + rho(i,j,k) = density_elem_buggy_Wright_loc( T(i,j,k), S(i,j,k), pressure(i,j,k)) +enddo +``` +— no `this`, no polymorphic dispatch inside the device loop at all. + +### 4.3 Intrinsic replacements — `cuberoot`/`nth_root` (`MOM_intrinsic_functions.F90`) + +`cuberoot` predates the GPU port (added by the upstream `dev-gfdl` history); the dev/gpu-specific +changes are (a) adding `!$omp declare target` to `cuberoot`, `rescale_cbrt`, and `descale` +(`dev-gfdl...dev/gpu` diff for this file — no separate commit message survives on `dev/gpu`'s +first-parent history, folded into `546907fee`/`7b706ecbc`), and (b) adding a **new** function, +`nth_root`, generalizing the pattern to arbitrary integer roots for `MOM_barotropic.F90`'s +`bt_rem = av_rem**Instep`. `nth_root`'s doc comment states the motivating problem precisely: + +```fortran +!> Bit-stable n-th root of x for x in (0, +inf) and integer n >= 1, suitable +!! for evaluation inside `!$omp target` / `do concurrent` offloaded regions. +!! +!! Lowering `x**(1.0/n)` via the compiler produces `exp((1.0/n)*log(x))` — two +!! transcendentals whose last-bit rounding differs between host libm and CUDA +!! libdevice. This routine avoids that path entirely: it uses fixed-iteration +!! Newton on y^n - x = 0, with y^(n-1) evaluated as repeated multiplication, +!! and one bit-precision-polishing iteration at the end. +elemental function nth_root(x, n) result(root) + !$omp declare target +``` +This is the pattern in miniature: identify an intrinsic (`x**(1/n)`) whose device lowering is not +bit-reproducible with the host, and replace it with a deterministic, fixed-iteration-count, +multiply/add-only kernel that is `elemental` (so it vectorizes/parallelizes trivially) and carries +`!$omp declare target` (so it's legal inside `!$omp target` regions too). + +### 4.4 `flux_elem`/`ratio_max` — already `elemental`/`pure`, just needed force-inlining (§3 above) + +Unlike 4.1–4.3, these were **already** `elemental`/`pure` before the GPU port (they long predate +`dev/gpu`); the fix here was not a restructuring of the arithmetic at all, just a force-inline +directive (or the build-flag equivalent) — which itself churned three times: `-Minline` flag +(`3cb184edd`) → `!NVF$ INLINE` (`4e3f1b758`) → `!DIR$ ATTRIBUTES FORCEINLINE` on the two `flux_elem` +routines, none on `ratio_max` (`93dbbd36e`; see §1 banner and §3). This is the cheapest end of the +spectrum: when the helper is already side-effect-free, the only remaining problem is *compiler code +generation* (inlining), not source structure. + +**Common thread across all four:** none of the fixes reorder floating-point operations or change +which numbers get added to which. `efp_decompose` moves code, not arithmetic. The EOS `_loc` split is +a literal copy-paste of the RHS with the `this` dummy dropped. `nth_root`/`cuberoot` replace a +compiler-lowered intrinsic with an equivalent-precision hand-written kernel (arithmetically different +from `x**(1/n)` at the bit level, but *designed* to match to round-off, and unit-tested via +`Test_cuberoot`). `flux_elem`/`ratio_max` are untouched. This is consistent with guiding principle #2 +in `00-architecture.md`: bitwise reproducibility is mandatory, and `pure`/`elemental` extraction is +the preferred restructuring tool precisely because it cannot silently reorder a reduction. + +--- + +## 5. Constructs nvfortran refused inside device regions + +### 5.1 Early `exit`/`return` inside a loop under `do concurrent` — commit `e23d6a7b1` + +``` +swap early exit to if guard in insert sort + +NVHPC 25.11 didn't like the early exit and would +give wrong answers. +``` +Diff, `src/tracer/MOM_tracer_hor_diff.F90:942-946` (insertion-sort inner loop, itself nested inside an +outer `do concurrent (j=js-1:je+1, i=is-1:ie+1)`): +```fortran + do concurrent (j=js-1:je+1, i=is-1:ie+1) + do k=2,num_srt(i,j) ; if (rho_srt(i,k,j) < rho_srt(i,k-1,j)) then + ! The last segment needs to be shuffled earlier in the list. +- do k2 = k,2,-1 ; if (rho_srt(i,k2,j) >= rho_srt(i,k2-1,j)) exit ++ do k2 = k,2,-1 ; if (rho_srt(i,k2,j) < rho_srt(i,k2-1,j)) then + itmp = k0_srt(i,k2-1,j) ; k0_srt(i,k2-1,j) = k0_srt(i,k2,j) ; k0_srt(i,k2,j) = itmp + tmp = rho_srt(i,k2-1,j) ; rho_srt(i,k2-1,j) = rho_srt(i,k2,j) ; rho_srt(i,k2,j) = tmp + tmp = h_srt(i,k2-1,j) ; h_srt(i,k2-1,j) = h_srt(i,k2,j) ; h_srt(i,k2,j) = tmp +- enddo ++ endif ; enddo + endif ; enddo + enddo +``` +The transformation negates the exit condition (`>=` → `<`) and turns it into a body `if`-guard that +still executes every iteration of the `k2` loop (doing nothing once the sorted position is found) +rather than jumping out of it early. Behaviorally identical on a CPU; the NVHPC 25.11 device code +generator for `do concurrent` mis-compiled the `exit`-from-inner-loop form and produced wrong values +— this is catalogued as a **compiler bug worked around**, not a language limitation (`exit` from +`do concurrent` bodies is otherwise permitted by the standard here since it doesn't cross the +`do concurrent` loop itself, only the ordinary nested `do`). + +### 5.2 `modulo()` — "not implemented on all systems" + +`MOM_intrinsic_functions.F90`'s `rescale_cbrt` avoids `modulo()` entirely, with the reasoning +recorded in-line: +```fortran + ! modulo() is not implemented on all systems, so compute the remainder as + ! r = n - 3*q. + + e_x = e_a - e_r * 3 +``` +This particular replacement (`850504a98`, "cuberoot: Replace modulo() with arithmetic ops") actually +originates upstream on `dev-gfdl` itself (authored by Marshall Ward, NOAA) — i.e. GFDL had already +hit non-`modulo()`-supporting platforms (explicitly named as "e.g. NVIDIA GPUs" in that commit +message) before `dev/gpu` forked, and `dev/gpu` inherited the fix along with the merge, then layered +`!$omp declare target` on top of the resulting `rescale_cbrt`/`descale`/`cuberoot`. The floor-division +identity used instead, `⌊e/3⌋ = (e + sign(1,e) - 1) / 3`, replaces `modulo(e,3)` with `sign()` + +integer truncating division, both of which are safe on device. + +### 5.3 Polymorphic dispatch / `select type` — the unresolved case + +Documented directly in-source (`MOM_EOS_Wright.F90:1008`, `:1048`, `:1114`, `:1147`; +`MOM_EOS_Roquet_rho.F90:817`) as an **open, unresolved** problem, distinct from the fixed cases above: +```fortran + ! NOTE: There is an implicit copy of `this` which cannot yet be prevented. + ! Possibly because Nvidia cannot associate `this` with `EOS%type`. + + if (present(rho_ref)) then + do concurrent (k=ks:ke, j=js:je, i=is:ie) + rho(i,j,k) = density_anomaly_elem_buggy_Wright(this, T(i,j,k), S(i,j,k), & + pressure(i,j,k), rho_ref) + enddo + else + do concurrent (k=ks:ke, j=js:je, i=is:ie) + rho(i,j,k) = density_elem_buggy_Wright_loc( T(i,j,k), S(i,j,k), pressure(i,j,k)) + enddo + endif +``` +Note the asymmetry within a single subroutine (`calculate_density_array_3d_buggy_Wright`, +`MOM_EOS_Wright.F90:1024`): the common (`rho_ref` absent) branch was converted to the `_loc` form and +avoids `this` entirely; the `rho_ref`-present branch **still passes `this`** into +`density_anomaly_elem_buggy_Wright`, which has not (yet) been given a `_loc` sibling, so it still +carries the "implicit copy of `this`" comment as an acknowledged, uncorrected cost. This is exactly +the case flagged unresolved in `00-architecture.md` §7.5. The root architectural problem — `EOS_type` +wraps `class(EOS_base), allocatable :: type` (`MOM_EOS.F90:162`) and dispatches via +`EOS%type%calculate_density_...` (`MOM_EOS.F90:344` et seq., chosen via `select type +(t => EOS%type)` at init, `MOM_EOS.F90:2143`) — remains in place for every EOS variant except +buggy-Wright and Roquet_rho, which is why the doc calls the EOS layer "being rewritten" rather than +"rewritten." + +### 5.4 `select type` at the dispatch boundary (not itself inside a device loop) + +Worth noting for completeness: the `select type (t => EOS%type)` construct in `MOM_EOS.F90:2143` +(used only during `EOS_init` to call type-specific parameter setters) is host-only and not itself a +device-region problem — it is the *runtime v-table dispatch through `class(EOS_base)`* at +per-gridpoint call sites (`EOS%type%density_elem(...)`, `EOS%type%calculate_density_array_3d(...)`) +that nvfortran cannot resolve on device, which is why the fix is at the call site (duplicate as a +free function) rather than at the type declaration. + +--- + +## 6. Where code is duplicated (not shared) to avoid a cross-module/cross-procedure device call + +| Duplication | Files | What's duplicated | Maintenance cost | +|---|---|---|---| +| `density_elem_buggy_Wright` / `density_elem_buggy_Wright_loc` | `MOM_EOS_Wright.F90:90-117` | Full arithmetic body duplicated verbatim between the type-bound wrapper (kept for API compatibility with polymorphic callers) and the free `_loc` function | Any bugfix or unit change to the Wright density formula must be applied in the `_loc` body; the wrapper is a 1-line pass-through so it can't drift on its own, but a future edit to one and not the other would silently split behavior between the polymorphic-dispatch call path and the device call path | +| `calculate_density_derivs_elem_buggy_Wright` / `..._loc` | `MOM_EOS_Wright.F90:199-241` | Same pattern for density-derivative arithmetic | Same risk, doubled: this is now the *third* copy of closely related Wright-EOS arithmetic (density, density anomaly (not yet split, still polymorphic-only), derivatives) living in the file, at different stages of the `_loc` migration | +| Roquet_rho density/derivs `_loc` siblings | `MOM_EOS_Roquet_rho.F90` (mirrors `MOM_EOS_Wright.F90` structure) | Same wrapper/`_loc` split, independently re-implemented per EOS formulation | Every additional EOS type that gets GPU-ported (currently only 2 of 9 `form_of_EOS` cases: buggy-Wright, Roquet_rho) needs its own from-scratch `_loc` split; there is no shared free-function template, so the split is copy-pasted per formulation rather than factored once | +| `find_L_open_uniform_slope` / `_concave_trigonometric` / `_concave_iterative` / `_convex` | `MOM_set_viscosity.F90:1241-1802` | Four independent bottom-shape-specific column kernels, each `!$omp declare target`, rather than one dispatching helper | This is a pre-existing physics branch structure (different closed-form/iterative solutions per bottom curvature case), not new duplication caused by porting — but the device-residency requirement means *all four* must independently carry the directive and be kept in sync if the calling convention changes, since there is no shared dispatch layer that could paper over an inconsistency | +| `find_coupling_coef` / `find_coupling_coef_k` / `find_coupling_coef_gl90` | `MOM_vert_friction.F90:436,2099,2609` | Three separate per-column coupling-coefficient kernels (BBL-drag, GL90, ice-shelf-drag variants) each independently marked `!$omp declare target` | Same shape as above: physics-driven branching pre-dates the port, but porting forces each branch to be independently verified device-safe; a future change to the shared thickness/coupling logic must be replicated three times unless a common inner helper is factored out (none currently exists) | + +**Overall maintenance cost pattern:** the `_loc`-suffix duplication (EOS) is the clearest *port-induced* +duplication — it exists solely because polymorphic `this` cannot cross into device code, and doubles +the number of functions that must be kept in sync per EOS formulation as more formulations are ported. +The `find_L_open_*`/`find_coupling_coef*` families are pre-existing physics-driven branch structures +that the port did not duplicate further, but whose per-branch `!$omp declare target` marking means +device-safety must be independently re-verified for each variant rather than centrally. + +--- + +## 7. Prescriptive rules — the device-call checklist + +Distilled from the cases above. **When a device compute region (`!$omp target`/`!$omp loop`/ +`do concurrent`) must call a helper, walk this checklist in order:** + +1. **Is the helper side-effect-free?** It must not touch module-level state, do I/O, or mutate + `save`d/`intent(inout)` module data. If not, refactor: extract the innermost arithmetic into a new + `pure` (or `elemental` for scalar-per-element) procedure and return errors/flags as arguments — + the `efp_decompose` model (`MOM_coms.F90:778`, side-effect-freedom is *why* it can sit in a + `do concurrent` reduction body). Never reorder the floating-point arithmetic while doing this + (guiding principle #2, `00-architecture.md`). +2. **Does it carry no `class(*)`/polymorphic dummy?** A `class(this)` argument forces an implicit copy + nvfortran mishandles on device (`MOM_EOS_Wright.F90:1048`, "implicit copy of `this` which cannot yet + be prevented"). If it has one, duplicate it as a free `_loc` function with the `this` dummy dropped + and the body copy-pasted verbatim (`density_elem_buggy_Wright_loc`, commit `52a1b3954`), and call + the `_loc` form from the loop. +3. **Which device-region idiom is the call inside?** + - *Bare `do concurrent`*: nvfortran's `-stdpar=gpu` lowering generally handles `pure`/`elemental` + calls without a mandatory `declare target` — the definition just has to be visible. + - *`!$omp target teams` / `!$omp target … !$omp loop`*: the callee **must** be device-resident — + either `!$omp declare target` on the definition, or force-inlined at compile (`!DIR$ ATTRIBUTES + FORCEINLINE :: ` at HEAD; historically `!NVF$ INLINE` + `-Minline=pragma`). Getting this + wrong is **silently wrong numbers, not a build error** (`3cb184edd`). +4. **Declare-target vs force-inline — which?** Column/point kernels that are large or reused widely get + `!$omp declare target` (all 21 catalogued helpers). Tiny leaf `elemental`/`pure` helpers that the + compiler *can* inline get force-inlined instead — this measurably outperformed out-of-line device + calls (`93dbbd36e`: FORCEINLINE "significantly improves performance … at -O2"). If you add `!$omp + declare target`, it must appear after all declarations and before the first executable statement + (the `MOM_set_viscosity.F90` routines even repeat it twice, `:1251`/`:1258` — harmless idiosyncrasy). +5. **Same-module or cross-module?** Irrelevant to the compiler: the boundary that triggers the + requirement is "outside the lexical scope of the enclosing `!$omp target` construct", not the + Fortran `module` boundary (§2 note). A private same-module helper called from a sibling subroutine + hits the identical problem as a genuine cross-module call. Duplicate-per-module only when forced + (the `_loc` split is re-implemented per EOS formulation — no shared template, §6). +6. **Verify.** There is no compile-time signal for the silent-wrongness failure mode. The only proof is + a **`MOM_checksums` hchksum/uchksum + reproducing-sum energy comparison, CPU build vs GPU build, + bit-for-bit** (`00-architecture.md` §7.2, `MOM_checksums.F90:2680`). A port that compiles and runs + but whose helper silently failed to inline will diverge only here. + +### The "never do inside a device region" list + +| Never | Why | Evidence | +|---|---|---| +| Pass a `class(*)`/polymorphic `this` into the loop | nvfortran emits an implicit device copy it mishandles → runtime errors | `MOM_EOS_Wright.F90:1008,1048,1114,1147`, `MOM_EOS_Roquet_rho.F90:817`; fix `52a1b3954`/`7c7af5572` | +| Dispatch through a v-table (`EOS%type%method(...)`) | nvfortran cannot resolve the `class(EOS_base)` v-table on device | `MOM_EOS.F90:162,2143`; §5.3/§5.4 | +| Early `exit`/`return` out of an inner loop under `do concurrent` | NVHPC 25.11 mis-compiles it → **wrong answers**; rewrite as a negated `if`-guard that still iterates | `e23d6a7b1`, `MOM_tracer_hor_diff.F90:942-946` | +| `modulo()` | not implemented on all device targets (NVIDIA GPUs); use `sign()` + truncating integer division | upstream `850504a98` (Marshall Ward/NOAA), `MOM_intrinsic_functions.F90:232-235` | +| `x**(1/n)` / `x**(1./3.)` where bit-reproducibility matters | lowered to `exp((1/n)*log(x))` — two transcendentals whose last bit differs between host libm and CUDA libdevice | `nth_root`/`cuberoot`, `MOM_intrinsic_functions.F90:120-132,50` | +| Call a helper into an `!$omp target teams`/`loop` without a *guaranteed* inline or `declare target` | silent wrong numbers, no build error | `3cb184edd` ("MANDATORY … otherwise results are incorrect") | +| Allocate, do I/O, or post a diagnostic inside the loop | forces host round-trips / is illegal on device | `00-architecture.md` §9 | + +--- + +## 8. Cross-references + +- `docs/gpu-knowledge/00-architecture.md` §0.4 (guiding principle), §5 (k-blocking hybrid kernel using + `flux_elem`/`ratio_max`), §7.1 (EOS polymorphism), §7.5 (compiler workarounds index). +- Planned/pending docs referenced but not yet written at time of writing: `06-eos.md` (full EOS + architecture), `13-compiler-workarounds.md` (broader nvfortran-bug catalogue including + `num_teams`/`thread_limit` tuning (`5b5f6b2b1`), `omp target teams loop` → `do concurrent` + reversions (`e8b0ecfbf`), and the A100/nvfortran-25.5 crash workaround (`2108e0eba`) — these are + compiler-workaround entries adjacent to but outside the cross-module-inlining scope of this + document. + +--- + +## Verification notes + +Verified against source at `dev/gpu` HEAD and git history (source + git only; no build/run). + +**Confirmed (verbatim / exact):** +- The **21 `!$omp declare target`** count and every catalogued def line: `MOM_intrinsic_functions.F90` + cuberoot `:50`/nth_root `:132`/rescale_cbrt `:180`/descale `:245`; `MOM_coms.F90` `declare + target(pr,I_pr)` `:69` and `efp_decompose` def `:778`/directive `:779`; `MOM_vert_friction.F90` + find_coupling_coef_gl90 `:436`, _k `:2099`, plain `:2609`; `MOM_set_viscosity.F90` 12 occurrences. +- The **double-directive idiosyncrasy** in `MOM_set_viscosity.F90` (e.g. `find_L_open_uniform_slope` + starts `:1241`, directives at `:1251` after dummy decls and `:1258` after local decls, both before + the first executable at `:1260`). 5×2 + 2×1 = 12 confirmed. +- Commit `3cb184edd` message ("inlining of ratio_max and flux_elem is MANDATORY … Otherwise results + are incorrect") — verbatim. Commit `4e3f1b758` (added `!NVF$ INLINE` to all 3, dropped + `thread_limit(128)`) — verbatim + diff. Commit `52a1b3954` message and the wrapper doc-comment + (`MOM_EOS_Wright.F90:107-109`). Commit `e23d6a7b1` message ("NVHPC 25.11 didn't like the early exit + and would give wrong answers") + diff (`MOM_tracer_hor_diff.F90:942-946`, `>=`→`<`, `exit`→guard). +- `modulo()` origin `850504a98` — confirmed upstream, author **Marshall Ward (NOAA)**, message names + "NVIDIA GPUs" and the `sign()` simplification; the in-source comment `:232-235`. +- The **5 "implicit copy of `this`"** occurrences: `MOM_EOS_Wright.F90:1008,1048,1114,1147` + + `MOM_EOS_Roquet_rho.F90:817`. The **asymmetric `_loc` conversion** in + `calculate_density_array_3d_buggy_Wright` (`:1024`): `rho_ref`-present branch still passes `this` + (`:1053`), absent branch uses `_loc` (`:1058`) — exact. +- The `nth_root` doc-comment (`exp((1/n)*log(x))` transcendental-lowering rationale, Newton iteration) + — verbatim (`:120-132`). +- **NVHPC version numbers are correct as differentiated:** `e23d6a7b1` = **25.11**; `2108e0eba` = + **25.5** (a *different* commit — "crashes on stellar A100s with nvfortran 25.5"). No conflict; both + right per their own commit. + +**Corrected (material):** +- **`!NVF$ INLINE` is stale.** At HEAD (after merged commit `93dbbd36e`, "remove nvf inline and + replace with intel forceinline") there are **zero** `!NVF$ INLINE` directives in `src/`. `flux_elem` + (`:1086`) and `flux_elem_OBC` (`:1149`) now carry `!DIR$ ATTRIBUTES FORCEINLINE :: `, and + **`ratio_max` carries no inline directive at all**. Rewrote §1 banner, §2 catalogue rows + total, + §3 (added the third-form subsection), and §4.4. +- **§1's `do concurrent` example was wrong:** it claimed "`ratio_max` inside a bare `do concurrent` at + `:1622-1630`". Those lines are `flux_elem` calls inside an `!$omp loop`; `ratio_max` is called at + `:768-769` etc. inside `!$omp target`/`!$omp loop`. Corrected. +- Minor: `efp_decompose` def is `:778` (directive `:779`), not `:779`; catalogue's `flux_elem` + call-site list mixed in `flux_elem_OBC` sites (`:724,1065,…`) — split out. + +**Confidence:** High. Every quoted commit message, doc comment, and line number was re-derived from +the tree. The former open item — how `ratio_max` stays correct with no inline directive and no +`declare target` while called from a device region — is resolved in §3: no `-Minline` flag is +covering for it, so correctness rests on nvfortran's implicit device codegen for a small same-file +`pure` function, which works on the tested toolchain but should be pinned with an explicit +FORCEINLINE. diff --git a/knowledge/gpu-knowledge/09-barotropic-solver.md b/knowledge/gpu-knowledge/09-barotropic-solver.md new file mode 100644 index 0000000..de39a2f --- /dev/null +++ b/knowledge/gpu-knowledge/09-barotropic-solver.md @@ -0,0 +1,557 @@ +# The Barotropic Solver — `MOM_barotropic.F90` (dev/gpu) + +> **Purpose.** Deep dive on `btstep`, the barotropic (fast-mode) sub-cycle that `MOM_dynamics_split_RK2.F90` +> calls twice per baroclinic step (predictor + corrector, `00-architecture.md` §4.2 step 6/12). At +> 6868 lines and 242 `do concurrent` / 106 `omp target` occurrences it is the single most +> directive-dense module on `dev/gpu`, and it is the only module using a **wide-halo domain** +> (`CS%BT_Domain`) to amortize MPI cost over many fast time steps. Read `00-architecture.md` §4.2–4.3 +> first. All line numbers below are against `src/core/MOM_barotropic.F90` on `dev/gpu` unless noted. + +--- + +## 1. Algorithm structure and the sub-cycle + +### 1.1 Call shape + +`btstep` (`:480`) is called from `MOM_dynamics_split_RK2.F90:726` (predictor) and `:1023` (corrector). +Its job: given baroclinic accelerations (`bc_accel_u/v`), layer velocities/pressure-force terms, and +the provisional free-surface `eta_in`, integrate the 2‑D (depth-averaged) shallow-water equations +through many small time steps `dtbt` to cover one full dynamics step `dt`, returning barotropic +accelerations (`accel_layer_u/v`), the final `eta_out`, and time-averaged transports (`uhbtav`, +`vhbtav`). Internally it: + +1. Sets up wide-halo copies of `eta`, Coriolis PV `q`, and various coefficient arrays (`:850`–`:1300` + region) — `linearized_BT_PV` branch pre-computes `q` from `CS%q_D` "quite early... to start the + halo update that needs to be completed before the next calculations" (comment at `:883`). +2. Calls `btstep_find_Cor` (`:3159`) to build the Coriolis coefficient arrays `f_4_u`/`f_4_v` used in + the C-grid Coriolis bracket. +3. Calls `btstep_ubt_from_layer` (`:3671`) to project the 3-D layer velocities down onto an initial + barotropic velocity (`ubt`, `vbt`) using the weights `wt_u`/`wt_v` (see §3 below for the + reproducibility bug in this weight calculation). +4. Calls `BT_cont_to_face_areas` / `set_local_BT_cont_types` to build the face-area closure + (`BT_cont_type`, see §6) used for the nonlinear continuity option `use_BT_cont`. +5. Calls `btstep_timeloop` (`:2376`) — **the actual sub-cycle**, described in §1.2. +6. Calls `btstep_layer_accel` (`:3723`) to project the resulting barotropic acceleration back onto + each layer, weighted by `visc_rem_u/v`, producing `accel_layer_u/v`. + +### 1.2 `btstep_timeloop` — why the wide halo matters + +`btstep_timeloop` (`:2376`) runs `do n=1,nstep+nfilter` (`:2751`, closing `:3143`), alternately +updating `eta`, then `u`/`v` (order alternates by parity: `v_first = (MOD(n+G%first_direction,2)==1)`, +`:2804`), then the transports `uhbt`/`vhbt`. Each of these local updates only needs a 1- or 2-point +stencil (`stencil = max(1, CS%min_stencil)`, `:2622`, bumped to 2 for nonlinear continuity with a +finite update period, `:2623-2624`). + +The key trick is in the "march inward" logic at `:2621-2630` and `:2754-2763`: + +```fortran +! Figure out the fullest arrays that could be updated. +stencil = max(1, CS%min_stencil) +... +num_cycles = 1 +if (CS%use_wide_halos) & + num_cycles = min((is-CS%isdw) / stencil, (js-CS%jsdw) / stencil) +isvf = is - (num_cycles-1)*stencil ; ievf = ie + (num_cycles-1)*stencil +jsvf = js - (num_cycles-1)*stencil ; jevf = je + (num_cycles-1)*stencil +... +do n=1,nstep+nfilter + ... + ! Update the range of valid points, either by doing a halo update or by marching inward. + if ((iev - stencil < ie) .or. (jev - stencil < je)) then + call do_group_pass(CS%pass_eta_ubt, CS%BT_Domain, clock=id_clock_pass_step, omp_offload=.true.) + isv = isvf ; iev = ievf ; jsv = jsvf ; jev = jevf + else + isv = isv+stencil ; iev = iev-stencil + jsv = jsv+stencil ; jev = jev-stencil + endif +``` +(`:2621-2630`, `:2754-2763`) + +Because `CS%BT_Domain` is cloned from `G%Domain` with a **much wider halo** than the normal `NIHALO_=2` +(`clone_MOM_domain(G%Domain, CS%BT_Domain, min_halo=wd_halos, symmetric=.true.)`, `:6104`; `wd_halos` +sized from `BT_halo_sz`/`use_wide_halos` params), a single MPI halo exchange at the top of the loop +(or none, if `num_cycles>1`) fills in enough valid data that the algorithm can run `num_cycles` +barotropic steps "marching inward" — shrinking the valid index range by `stencil` points per step — +before it runs out of valid halo data and must do another real halo exchange. This is exactly why the +barotropic solver, whose per-step physics is trivial (shallow-water update), doesn't become +communication-bound even though it may sub-cycle 10s of steps per baroclinic step: `num_cycles` steps +run for the cost of one exchange. `CS%isdw/iedw/jsdw/jedw` (set at `:6125-6126`) are the wide-halo +bounds; `BT_USE_WIDE_HALOS` (param, `:5817`) toggles the feature and `BT_HALO` sizes the halo. + +### 1.3 `set_dtbt` — choosing the barotropic step + +`set_dtbt` (`:3797`) computes `CS%dtbt`, the sub-step length, from a CFL-type stability estimate. +Given `pbce` (baroclinic pressure-anomaly sensitivity) or a rough `gtot_est`, it builds `gtot_E/W/N/S` +(the effective reduced gravity felt at each face, `:3880-3900`), combines them with face areas +`Datu`/`Datv` (from `BT_cont_to_face_areas` or `find_face_areas`) and grid metrics into a local +squared-timestep bound (`:3903-3912`): + +```fortran +do concurrent (j=js:je, i=is:ie) DO_LOCALITY(reduce(min:min_max_dt2)) + Idt_max2 = 0.5 * (1.0 + 2.0*CS%bebt) * (G%IareaT(i,j) * & + (((gtot_E(i,j)*Datu(I,j)*G%IdxCu(I,j)) + (gtot_W(i,j)*Datu(I-1,j)*G%IdxCu(I-1,j))) + & + ((gtot_N(i,j)*Datv(i,J)*G%IdyCv(i,J)) + (gtot_S(i,j)*Datv(i,J-1)*G%IdyCv(i,J-1)))) + & + ((G%Coriolis2Bu(I,J) + G%Coriolis2Bu(I-1,J-1)) + & + (G%Coriolis2Bu(I-1,J) + G%Coriolis2Bu(I,J-1))) * CS%BT_Coriolis_scale**2 ) + if (Idt_max2 * min_max_dt2 > 1.0) min_max_dt2 = 1.0 / Idt_max2 +enddo +``` +(`:3903-3912`) — a `do concurrent` with a `DO_LOCALITY(reduce(min:...))` clause (the GPU-portable +min-reduction idiom, see `00-architecture.md` §9), followed by a **global** `min_across_PEs(dtbt_max)` +(`:3916`, an MPI allreduce — the one place per baroclinic step where all PEs must synchronize on the +barotropic step size). `CS%dtbt = CS%dtbt_fraction * dtbt_max` (`:3919`), where `dtbt_fraction` +defaults to 0.98 (or the negative of param `DTBT` if the user supplies a safety fraction; `:6404`). +Back in `btstep` (`:807`): `nstep = CEILING(dt/CS%dtbt - 0.0001)` — the number of barotropic steps +needed to cover the full baroclinic `dt`. `set_dtbt` is called once at `barotropic_init` (`:6581`) to +get a startup estimate and once per baroclinic step from `MOM_dynamics_split_RK2.F90:715/719`; if +`DTBT>0` (fixed timestep) it's skipped at runtime and `dtbt` is just read from the input parameter or +restart file (`:6583-6588`). + +Note `set_dtbt` brackets its `gtot_*`/`Datu`/`Datv` scratch arrays in explicit +`!$omp target enter data map(alloc: ...)` / `map(release: ...)` (`:3864`, `:3913`) rather than relying +on `barotropic_init`-time mapping, since these are transient per-call locals, not CS members. + +--- + +## 2. GPU status on mainline `dev/gpu` + +**Directive density:** 242 `do concurrent`, 106 `omp target` in this one file — the highest raw count +of any ported module (`00-architecture.md` §4.3, §6.1: `+1003/−804` lines vs `dev-gfdl`). `git log +--oneline dev-gfdl..dev/gpu -- src/core/MOM_barotropic.F90` shows **100 commits** touching this file, +almost entirely small, single-purpose "port this loop" / "alloc this array" commits (`c2b162eda +btcalc: promote hatu to 3d` → ... → `7616a34c5 btstep: port btstep_ubt_from_layer subroutine` → +`65cf95a6f btstep: port btstep_find_Cor` → `1159cbd23 btstep: port eta_out loop` → +`8c62b0d37 send wt_* to gpu for btstep_timeloop` → `9013e8514 kji -> jki some loops` → +`e68e0e83a Dycore: Move halo updates to GPU`), i.e. the module was ported loop-by-loop and +array-by-array over many small, individually-reviewable commits rather than one large rewrite — +consistent with the "bitwise reproducibility is mandatory" guiding principle (`00-architecture.md` §0.2). + +### 2.1 `do concurrent` vs `omp target` + +- `do concurrent` is the default idiom for all the elementwise/columnwise updates inside `btstep`, + `btcalc`, `bt_mass_source`, `btstep_timeloop`'s per-step update, `btstep_find_Cor`, + `btstep_ubt_from_layer`, and `btstep_layer_accel` — e.g. the velocity update loops + `btloop_update_u`/`btloop_update_v` (ported by commits `063f8da03`/`70b25b3c3`) and the pressure-force + loop `btloop_find_PF` (`c2c4d4670`). +- `!$omp target` appears almost exclusively as **data-movement directives** + (`target enter/exit data`, `target update to/from`) around the wide-halo scratch arrays and CS + members, plus the 11 `do_group_pass(..., omp_offload=.true.)` calls (§2.2) and a few explicit + `!$omp target data`/`!$omp end target data` regions bracketing `btcalc`'s per-column loop (see the + `5f413739b` diff, §3). There is essentially no `!$omp target teams`/`loop` compute offload in this + file — unlike `MOM_continuity_PPM.F90`'s k-blocked hybrid kernels (`00-architecture.md` §5), btstep + relies on `do concurrent` alone for compute and on OpenMP purely for host/device data-transfer + bookkeeping. + +### 2.2 The ~11 group passes, all `omp_offload=.true.` + +`grep -n omp_offload src/core/MOM_barotropic.F90`: + +| Line | Pass | Domain | Purpose | +|---|---|---|---| +| `:1008` | `CS%pass_q_DCor` | `CS%BT_Domain` | wide-halo update of PV `q`/`DCor_u`/`DCor_v` before Coriolis-coefficient calc (clock `id_clock_pass_pre`) | +| `:1550` | `CS%pass_gtot` | `CS%BT_Domain` | wide-halo update of `gtot_E/W/N/S` | +| `:1551` | `CS%pass_ubt_Cor` | `G%Domain` | normal-halo update of reference Coriolis velocities | +| `:1772` | `CS%pass_eta_bt_rem` | `CS%BT_Domain` | wide-halo update of `eta`, `bt_rem_u/v`, `Rayleigh_u/v`, etc. | +| `:1773` | `CS%pass_Dat_uv` | `CS%BT_Domain` | face-area `Datu`/`Datv` (only when `.not. use_BT_cont`) | +| `:1774` | `CS%pass_force_hbt0_Cor_ref` | `CS%BT_Domain` | `BT_force_u/v`, `uhbt0/vhbt0`, `Cor_ref_u/v` | +| `:2022` | `CS%pass_e_anom` | `G%Domain` | eta anomaly used for time-averaged diagnostics | +| `:2070` | `CS%pass_ubta_uhbta` | `G%Domain` | time-averaged `CS%ubtav/vbtav`, `uhbtav/vhbtav` (the `omp_offload=.true.` was added in `9f4e48be1` "offload pass_uta_uhbta") | +| `:2757` | `CS%pass_eta_ubt` | `CS%BT_Domain` | **the inner sub-cycle halo pass** inside `btstep_timeloop`'s `do n=1,nstep+nfilter` loop (clock `id_clock_pass_step`) — this is the one amortized by wide halos, §1.2 | +| `:5290` | `BT_cont%pass_polarity_BT` | `BT_Domain` | face-polarity for the BT_cont closure | +| `:5291` | `BT_cont%pass_FA_uv` | `BT_Domain` | face areas `FA_u_*`/`FA_v_*` for the BT_cont closure | + +All 11 pass `omp_offload=.true.` so the halo exchange operates on device-resident buffers without a +round-trip to host (`00-architecture.md` §7.3, `do_group_pass` optional arg, +`config_src/infra/FMS2/MOM_domain_infra.F90:1143`). This is the same count and same call sites on the +`edoyango/acc-btstep` experimental branch (§5) — that branch does not touch the halo-exchange strategy, +only the compute-kernel scheduling around it. + +### 2.3 CS array members mapped in `barotropic_init` (`:6579-6676`) + +Representative CS-member enter-data calls, in the order they appear at the end of `barotropic_init`: + +```fortran +!$omp target enter data map (to: CS%frhatu, CS%frhatv) +!$omp target enter data map (to: CS%eta_cor) +... +!$omp target enter data map(to: CS%bathyT) +!$omp target enter data map(to: CS%D_u_Cor, CS%D_v_Cor) +!$omp target enter data map(to: CS%dx_Cv, CS%dy_Cu) +!$omp target enter data map(to: CS%IareaT, CS%IareaT_OBCmask) +!$omp target enter data map(to: CS%IDatu, CS%IDatv) +!$omp target enter data map(to: CS%IdxCu, CS%IdyCv) +!$omp target enter data map(to: CS%OBCmask_u, CS%OBCmask_v) +!$omp target enter data map(to: CS%q_d) +!$omp target enter data map(to: CS%ua_polarity, CS%va_polarity) +!$omp target enter data map(to: CS%ubtav, CS%vbtav) +``` +(`:6579-6580`, `:6667-6676`) + +`CS%frhatu`/`CS%frhatv` (`:118-121`) are the fraction-of-column-thickness-per-layer arrays computed by +`btcalc` and read every barotropic step by `set_dtbt` and the layer-projection routines; +`CS%eta_cor`/`CS%D_u_Cor`/`CS%D_v_Cor`/`CS%q_d` (i.e. `CS%q_D`) are all **wide-halo** arrays +(`isdw:iedw,jsdw:jedw` bounds, allocated at `:6135-6142`, `:6171-6180`, `:6308-6314`) — these are the +static/near-static per-timestep-invariant fields (bathymetry-derived depths, planetary vorticity) that +the sub-cycle reads every barotropic step without re-deriving, so they are mapped once at init and +persist device-resident for the life of the run. `CS%ubtav`/`CS%vbtav` are the running +time-averaged barotropic velocities accumulated across the sub-cycle (`:2954`, `:2959`) and consumed +by `barotropic_get_tav` (`:6681`) for the next predictor step's reference Coriolis velocities. + +--- + +## 3. The `frhat[uv]` HYBRID repro fix (`5f413739b`) + +Commit `5f413739b` ("Barotropic: frhat[uv] HYBRID repro fix", Marshall Ward) fixes **a bitwise +reproducibility regression under ifort**, not nvfortran — but it's directly relevant to the GPU port +because the loop structure it touches is exactly the kind of "split a loop for parallelism" refactor +the port does everywhere. `btcalc`'s `HYBRID` interpolation scheme (u-face thickness interpolation) +had been split into two separate `do concurrent` loops: one computing `CS%frhatu(I,j,k)` per layer, +and a second, separate loop summing `hatutot(I,j) = hatutot(I,j) + CS%frhatu(I,j,k)` across layers. +The fix folds the sum back into the same loop nest as the per-layer computation: + +```diff + CS%frhatu(I,j,k) = wt_arith*h_arith + (1.0-wt_arith)*h_harm + endif + endif +- enddo +- enddo +- !$omp end target data +- do concurrent (j=js:je, I=is-1:ie) +- do k=1,nz + hatutot(I,j) = hatutot(I,j) + CS%frhatu(I,j,k) + enddo + enddo ++ !$omp end target data +``` +(same pattern mirrored for `frhatv`). The commit message explains: "this patch fixes a minor bit +reproducibility regression in the calculation of `hat[uv]tot` and, consequently, `frhat[uv]`... it +seems possible that Intel has added a reduction-like optimization, even at `-O0`... this was notably +subtle, since many tests only call `btcalc()` once without the BTCONT `h_[uv]` inputs, was only +observed in `frhatu`, and did not actually change solution answers. Nonetheless, this is a genuine +answer change in an actively used solver, so we want to preserve bit repro until discussed and +approved by the consortium." The underlying lesson for the GPU port: **splitting a reduction-style +accumulation loop (`hatutot`) away from the loop that produces its addends is not always a +transparent refactor** — a compiler (ifort here, but nvfortran is equally capable of it) can reorder +or vectorize the split accumulation loop differently than it would the fused one, changing rounding. +The fix re-fuses the two loops rather than trying to force a specific evaluation order in the split +form, and the accompanying `!$omp end target data` scoping moved with it (the `target data` region +now wraps the single fused loop instead of the first of the two split ones). + +--- + +## 4. nvfortran-specific issues + +### 4.1 A100 / nvfortran 25.5 crash — `2108e0eba` + +Commit `2108e0eba` ("Remove eta_bt transfer from find_eta_2d that was causing crashes on stellar A100s +with nvfortran 25.5", Utheri Wagura) touches `src/core/MOM_interface_heights.F90`, not +`MOM_barotropic.F90` itself, but `find_eta`/`find_eta_2d` is called from the barotropic-adjacent SSH +diagnostics path in `MOM.F90`'s `step_MOM` (the `CS%eta_av_bc` / `ssh` bookkeeping that consumes the +barotropic solver's time-averaged output) and is one of the few remaining `target enter data ... if +(present(...))` constructs in the code, so it's catalogued here as a directly relevant nvfortran +compiler-bug workaround for code immediately downstream of `btstep`. The whole +`!$omp target enter data map(to: eta_bt) if (present(eta_bt))` directive — added a few weeks earlier +in `7003282d3` ("find_eta: Port to GPU and wrap calls") — was simply deleted: + +```diff + dZ_ref = 0.0 ; if (present(dZref)) dZ_ref = dZref + +- !$omp target enter data map(to: eta_bt) if (present(eta_bt)) +- + if (GV%Boussinesq) then + if (present(eta_bt)) then + do concurrent (j=js:je, i=is:ie) +``` +(`src/core/MOM_interface_heights.F90:247-249`, commit `2108e0eba`). The bug: an `if (present(...))` +clause guarding a `map(to:)` on an **optional dummy argument** crashed nvfortran 25.5 specifically on +Stellar's A100 GPUs. There's no replacement directive — the workaround is simply "don't map it there"; +presumably the caller already ensures `eta_bt` is mapped before the call (all six `find_eta` call +sites touched by `7003282d3` bracket the call with their own `!$omp target enter data +map(alloc:)`/`exit data map(from:)` around the whole call, e.g. +`MOM_ALE.F90:492-496`, `MOM.F90:1073-1077`). This is a clean example of "document nvfortran bugs... +genuine compiler bugs are hit and worked around" (`00-architecture.md` §0.5) — the fix is a deletion +with no compensating logic, purely because the conditional-map construct itself was what crashed. + +### 4.2 Checksum-transfer discipline — `b29b27150` + +Commit `b29b27150` ("btstep: Update GPU checksum transfers", Marshall Ward) is not a bug workaround +but shows the debugging-transfer pattern that has to be maintained by hand as fields move to/from +device: every `CS%debug`-gated `Bchksum`/`uvchksum`/`hchksum` call inside `btstep` needs an explicit +`!$omp target update from(...)` immediately before it, because the checksum routines run on the host +and read whatever is in the host-side copy of the array — which is stale once the corresponding +device buffer has diverged. The commit adds these one at a time to calls that had been missed: + +```diff + if (CS%linearized_BT_PV) then ++ !$omp target update from(CS%q_D) + call Bchksum(CS%q_D, "BT PV (q_D)", ...) + else ++ !$omp target update from(q) + call Bchksum(q, "BT PV (q)", ...) + endif ++ !$omp target update from(DCor_u, DCor_v) + call uvchksum("BT DCor_[uv]", DCor_u, DCor_v, ...) ++ !$omp target update from(Cor_ref_u, Cor_ref_v) + call uvchksum("BT Cor_ref_[uv]", Cor_ref_u, Cor_ref_v, ...) ++ !$omp target update from(uhbt0, vhbt0) + call uvchksum("BT [uv]hbt0", uhbt0, vhbt0, ...) + ... ++ !$omp target update from(visc_rem_u, visc_rem_v) + call uvchksum("BT visc_rem_[uv]", visc_rem_u, visc_rem_v, ...) ++ !$omp target update from(bc_accel_u, bc_accel_v) + call uvchksum("BT bc_accel_[uv]", bc_accel_u, bc_accel_v, ...) ++ !$omp target update from(CS%IDatu, CS%IDatv) + call uvchksum("BT IDat[uv]", CS%IDatu, CS%IDatv, ...) +``` +(`:1815-1850` region, commit `b29b27150`). This is not a compiler bug — it is a recurring maintenance +tax of the port: **every** debug/verification code path (`00-architecture.md` §9 "Verify a port") +has to be re-audited whenever a variable's residency changes, since a missed `target update from` will +silently checksum stale host data and can mask a real divergence (or manufacture a false one). Given +how frequently `btstep` was touched (100 commits), checksum transfers evidently drifted out of sync +with the mapping state repeatedly, hence a dedicated cleanup commit. + +--- + +## 5. The `edoyango/acc-btstep` OpenACC + async experiment + +`remotes/edoyango/acc-btstep` (2 commits ahead of `dev/gpu` on this file: `5bd8bb66d "add acc kernels +loop"`, `1ff44b2c5 "add asyncs"`; `git diff --stat dev/gpu...remotes/edoyango/acc-btstep -- +src/core/MOM_barotropic.F90` = `+289/−50`) is an **additive, non-competing** experiment: it does not +touch the `do concurrent` bodies, the `omp_offload=.true.` halo passes (identical 11 call sites, +verified by diff — same line-for-line `do_group_pass(..., omp_offload=.true.)` calls), or the +overall algorithm. It layers `!$acc kernels loop` directives *around* the existing `do concurrent` +loops and assigns them to one of **three OpenACC async queues** (`async(1)`, `async(2)`, `async(3)` — +counts on the branch: 185 `!$acc kernels`, 177 `async(`, 40 `!$acc wait`), with explicit `!$acc wait` +/ `!$acc wait(N)` barriers inserted wherever a downstream consumer (a halo pass, an `!$omp target +update`, an OBC calculation) needs the result: + +```fortran +! Zero out various wide-halo arrays. +!$acc kernels loop collapse(2) async(1) +do concurrent (j=CS%jsdw:CS%jedw, i=CS%isdw:CS%iedw) + gtot_E(i,j) = 0.0 ; gtot_W(i,j) = 0.0 + gtot_N(i,j) = 0.0 ; gtot_S(i,j) = 0.0 + eta(i,j) = 0.0 ; eta_PF(i,j) = 0.0 + ! ... (also eta_PF_1/d_eta_PF, eta_IC, dyn_coef_eta under their flags) +enddo +!$acc kernels loop collapse(2) async(2) +do concurrent (j=CS%jsdw:CS%jedw, I=CS%isdw-1:CS%iedw) + Cor_ref_u(I,j) = 0.0 ; BT_force_u(I,j) = 0.0 ; ubt(I,j) = 0.0 + Datu(I,j) = 0.0 ; bt_rem_u(I,j) = 0.0 ; uhbt0(I,j) = 0.0 +enddo +!$acc kernels loop collapse(2) async(3) +do concurrent (J=CS%jsdw-1:CS%jedw, i=CS%isdw:CS%iedw) + Cor_ref_v(i,J) = 0.0 ; BT_force_v(i,J) = 0.0 ; vbt(i,J) = 0.0 + Datv(i,J) = 0.0 ; bt_rem_v(i,J) = 0.0 ; vhbt0(i,J) = 0.0 +enddo +``` +(`:1018-1044`, commit `1ff44b2c5`), and later, before something that depends on all three streams: + +```fortran +!$acc wait +if (id_clock_calc_pre > 0) call cpu_clock_end(id_clock_calc_pre) +if (nonblock_setup) then + !$omp target update from(q, DCor_u, DCor_v) +``` +(`:1006`-area). The pattern extends into the hot sub-cycle: 35 `!$acc kernels`/`async(` occurrences +and 12 `!$acc wait`s fall inside `btstep_timeloop` itself, so the experiment reaches the actual +per-barotropic-step loop, not just the one-time setup section of `btstep`. + +**What this adds over mainline:** on `dev/gpu`, `do concurrent` loops are scheduled by whatever the +nvfortran runtime's default heuristic picks (typically one CUDA stream, synchronous-looking from +Fortran's point of view even though `do concurrent` has no sequencing guarantee by the standard). The +ACC branch takes explicit control: independent zero-init / setup loops that don't depend on each +other (three separate wide-halo zero-init loops — one h-point group, one u-point group, one v-point +group, in the example above) are put on **three +different async queues** so the GPU can overlap their kernel launches/execution instead of the runtime +serializing them, with `!$acc wait` inserted only at true data dependencies (a halo pass, an OBC +calc, a `target update`). + +**What it suggests about the current path's performance:** the fact that this branch exists at all — +and that it required manually auditing dozens of small independent loops in `btstep`'s setup and +sub-cycle to assign them to 3 concurrent streams — implies the author (Ed Yang) suspected the mainline +`do concurrent`-only approach was leaving overlap opportunities on the table: nvfortran's default +scheduling of `do concurrent` may serialize logically-independent kernels that don't share data, so +small, independent setup loops (of which `btstep` has many, e.g. the three wide-halo zero-init loops at +`:1018-1044`) pay full kernel-launch latency serially rather than overlapping. Because `btstep_timeloop`'s +inner loop is itself a chain of *dependent* steps (eta update → PF → Coriolis → velocity update → +transport, each step consuming the last), the value of async queues there is more about overlapping +the small independent per-step housekeeping (multiple wide-halo array zero/copy operations at the top +of each iteration) than about restructuring the sequential physics — consistent with `!$acc wait` +appearing right before each real dependency point rather than only at the end of the subroutine. No +performance numbers accompany either commit (no build/run per this study's constraints), so this +remains a plausible-but-unquantified hypothesis: OpenACC's explicit async model is being tried as a +finer-grained alternative to relying on `do concurrent`'s implicit (and possibly conservative) +scheduling, on the same underlying computation. + +**Evidence discipline (what the diff does and does not prove).** What is *verified from source*: the +branch is purely additive (`+289/−50`, no `do concurrent` body or halo-pass call site changed — the 11 +`omp_offload=.true.` sites are byte-identical), it introduces 185 `!$acc kernels loop`, 177 `async(N)` +clauses over queues 1/2/3, and 40 `!$acc wait`s, of which 35 kernels and 12 waits land inside +`btstep_timeloop` (`:2528-3350` on the branch). What is *not* in the diff: any benchmark, timing, or +profile. So the claim "nvfortran serializes independent `do concurrent` kernels and async recovers the +overlap" is an *inference from the shape of the intervention* (someone bothered to hand-assign queues), +not a measured result. State it as a hypothesis, not a finding. + +> **Reviewed 2026-07-14 (open):** NVHPC's documented model launches `do concurrent` kernels on the +> default CUDA stream per host thread, so serialization of independent kernels is *expected* — which +> is exactly what `acc-btstep`'s `async(1..3)` queues attack. Confidence is high, but quantify with +> one `nsys` timeline of btstep before investing. See KNOWLEDGE.md §9. + +--- + +## 6. Device-resident barotropic state: what and how + +Two categories of state must be device-resident for `btstep` to run without excessive host↔device +traffic: + +1. **`BT_cont_type`** (`MOM_variables.F90:317`) — the barotropic face-area closure. All-allocatable + members (`FA_u_EE/E0/W0/WW`, `uBT_WW/EE`, `FA_v_NN/N0/S0/SS`, `vBT_SS/NN`, optional `h_u`/`h_v`). + `alloc_BT_cont_type` (`MOM_variables.F90:567`) maps the **struct pointer itself** first + (`!$omp target enter data map(to: BT_cont)`, `:582`) then each member array individually right after + its `allocate(..., source=0.0)` (`:583-591`, `:593-601`, `:603-607`) — the member-by-member + "attach" pattern flagged as expensive in `00-architecture.md` §2.3 (cf. commit `1865612de`'s + flat-array refactor in `MOM_tracer_hor_diff` for the same reason). `BT_cont` is threaded through + `btstep`'s call signature as a `pointer` dummy argument (`:537`) and consumed by + `BT_cont_to_face_areas`/`set_local_BT_cont_types`, with its own wide-halo group passes + `pass_polarity_BT`/`pass_FA_uv` (`:5279-5286`, both `omp_offload=.true.` at `:5290-5291`). + +2. **Wide-halo `barotropic_CS` arrays** — the "always allocated with symmetric memory and wide halos" + locals declared at `:593-620` inside `btstep` (`q`, `ubt`, `bt_rem_u`, `BT_force_u`, `u_accel_bt`, + `uhbt`, `uhbt0`, `Cor_ref_u`, `Rayleigh_u`, `DCor_u`, `Datu`, and their v-counterparts) plus the + persistent CS members with `isdw:iedw,jsdw:jedw`-type bounds: `CS%bathyT`, `CS%IareaT`, + `CS%IareaT_OBCmask`, `CS%IdxCu`/`CS%IdyCv`, `CS%dx_Cv`/`CS%dy_Cu`, `CS%OBCmask_u`/`CS%OBCmask_v`, + `CS%D_u_Cor`/`CS%D_v_Cor`, `CS%q_D`, `CS%ua_polarity`/`CS%va_polarity` — all mapped once at the tail + of `barotropic_init` (`:6667-6676`, listed in full in §2.3) and never re-mapped per call, since they + are either static grid-derived metrics or run-persistent accumulators. The per-call locals (`q`, + `ubt`, `Datu`, etc., declared inside `btstep`/`btstep_timeloop`) instead get their own + `!$omp target enter data map(alloc: ...)` blocks scattered through `btstep` + (e.g. `:2663-2665`, `:2694-2695`) that are torn down before return, since their contents don't need + to persist across baroclinic steps — only within one `btstep` invocation. `CS%frhatu`/`CS%frhatv` + (the per-layer thickness fractions computed once per baroclinic step by `btcalc`, consumed every + barotropic sub-step by the layer-projection routines and by `set_dtbt`) sit in between: they persist + across the barotropic sub-cycle but are recomputed once per baroclinic step, and are mapped with + `to:` at `barotropic_init` (`:6579`, initial allocation) then refreshed via ordinary device-resident + writes inside `btcalc` each call (no repeated host round-trip). + +The unifying design point: **`CS%BT_Domain` (the wide-halo domain clone) is itself just a `MOM_domain_type` +pointer** (`:338`) — it carries no device-mapped array data of its own; what has to be device-resident +is every array whose valid range is described relative to its wide bounds (`CS%isdw/iedw/jsdw/jedw`), +because those are exactly the arrays the "march inward" trick (§1.2) reads and writes across many +barotropic steps between the relatively rare `do_group_pass(..., CS%BT_Domain, omp_offload=.true.)` +calls — if any one of them silently fell back to host residency, every barotropic sub-step touching it +would force a device→host→device round trip, defeating the entire wide-halo optimization. + +--- + +## 6.5 Transferable lessons for a porting agent + +Distilled from the commits above; each is a rule you can carry to the next module, with the barotropic +evidence that grounds it. + +1. **Loop-fusion is a reproducibility decision, not just a performance one.** When you split a loop to + expose parallelism, any accumulation you carry out of it (`hatutot += frhatu`, a running sum, a + dot-product) becomes a *separate* reduction the compiler is free to re-associate — even at `-O0`, + even under ifort (`5f413739b`). If the original fused loop set the bit pattern, keep the accumulate + inside that loop nest; do not "clean up" by hoisting it into its own `do concurrent`. The repro fix + is re-fusion, never a directive that pins evaluation order in the split form. Rule of thumb: **a + producer loop and the reduction that consumes its outputs must stay fused unless you have re-verified + the checksum after splitting them.** + +2. **Never `map(...) if (present(optional_arg))`.** A conditional data-map keyed on an optional dummy + argument crashed nvfortran 25.5 on A100 (`2108e0eba`). The fix is deletion, not repair — push the + mapping responsibility to the caller, which already brackets the call with its own + `enter data map(alloc:)/exit data map(from:)` around the whole callee. General pattern: **map + optional-argument buffers at the call site where presence is unambiguous, never inside the callee on + an `if (present(...))` guard.** + +3. **Every debug/verify path is a device→host transfer you must maintain by hand.** Checksums, `chksum0`, + and `[uv]/hchksum` run on the host and read the *host* copy; once a field is device-resident its host + copy is stale, so each debug call needs a matching `!$omp target update from(...)` immediately before + it (`b29b27150`). This drifts constantly (a module touched 100 times will silently checksum stale + data somewhere), so audit it whenever a variable's residency changes — a missing transfer both hides + real divergence *and* manufactures false ones. + +4. **Wide-halo device residency is all-or-nothing.** The "march inward" trick (§1.2) only pays off if + *every* array indexed on the wide bounds (`CS%isdw:iedw,jsdw:jedw`) is device-resident across the + whole sub-cycle. Static grid-derived metrics and run-persistent accumulators are mapped once at + `barotropic_init` and never re-mapped (§2.3); per-call scratch gets `enter/exit data map(alloc:)` + scoped to one `btstep`. A single wide-halo array that silently falls back to host residency turns + every one of the many barotropic sub-steps that touches it into a host round-trip — defeating the + entire optimization. When you port a wide-/deep-halo solver, treat "prove every halo-scoped array is + mapped" as a checklist item, not an afterthought. + +5. **Split the directives by job: `do concurrent` for compute, OpenMP `target` for data.** In this file + OpenMP is used almost exclusively for data movement (`target enter/exit data`, `target update`) and + for the `omp_offload=.true.` halo passes; the actual elementwise/columnwise math is `do concurrent` + (§2.1). Reductions are the documented exception — `do concurrent (...) DO_LOCALITY(reduce(min:...))` + in `set_dtbt` (§1.3), followed by an MPI `min_across_PEs`. This is the opposite of + `MOM_continuity_PPM`'s k-blocked `!$omp target teams` compute kernels (`00-architecture.md` §5): the + barotropic per-step physics is a cheap 2-D stencil, so it needs no manual team tuning, whereas + continuity's per-column reconstruction did. **Pick the compute idiom by kernel arithmetic intensity, + and keep OpenMP for the residency/transfer bookkeeping either way.** The `acc-btstep` experiment (§5) + is a *third* axis — explicit async scheduling — layered on top without disturbing either split. + +## 7. Cross-references + +- `00-architecture.md` §4.2 (call sequence into/out of `btstep`), §4.3 (directive-count summary), §7.3 + (halo/`omp_offload` infrastructure), §7.5 (compiler-workaround catalogue — the A100/nvfortran-25.5 + bug belongs there too). +- `03-openmp-mapping.md` for the general CS-member enter-data/exit-data lifecycle pattern that + `barotropic_init`/`barotropic_end` follow. +- Branches: `remotes/origin/btstep-halo-control`, `remotes/origin/bbl-cleanup-almost-step` (adjacent, + not reviewed here — halo-control and BBL cleanup respectively); `remotes/edoyango/acc-btstep` (§5). + +--- + +## Verification notes + +Verified against `src/core/MOM_barotropic.F90`, `src/core/MOM_interface_heights.F90`, +`src/core/MOM_dynamics_split_RK2.F90`, `src/core/MOM_variables.F90`, and git history on `dev/gpu` +(source + git only; no build/run, per constraints). + +**Confirmed exactly:** +- File scale and directive density: 6868 lines, 242 `do concurrent`, 106 `omp target`, 100 commits + touching the file on `dev-gfdl..dev/gpu`. +- All 11 `omp_offload=.true.` group-pass call sites and their line numbers (`:1008, :1550, :1551, + :1772, :1773, :1774, :2022, :2070, :2757, :5290, :5291`) — table in §2.2 is byte-accurate. +- `btstep` `:480`, called from `MOM_dynamics_split_RK2.F90:726/:1023`; `btstep_timeloop` `:2376`, + `btstep_find_Cor` `:3159`, `btstep_ubt_from_layer` `:3671`, `btstep_layer_accel` `:3723`, + `barotropic_get_tav` `:6681`. +- `num_cycles`/march-inward math at `:2621-2630`/`:2754-2763`, `v_first` at `:2804`, `stencil` logic — + code matches the doc's quoted block. +- `set_dtbt` (`:3797`): `reduce(min:min_max_dt2)` `do concurrent` at `:3903`, `min_across_PEs(dtbt_max)` + at `:3916`, `CS%dtbt = CS%dtbt_fraction*dtbt_max` at `:3919`, `map(alloc:)`/`map(release:)` at + `:3864`/`:3913`, `nstep = CEILING(dt/CS%dtbt - 0.0001)` at `:807`, `dtbt_fraction=0.98` at `:6404`; + called at `barotropic_init:6581` and `MOM_dynamics_split_RK2.F90:715/:719`. +- `clone_MOM_domain(..., min_halo=wd_halos, symmetric=.true.)` at `:6104`; `BT_USE_WIDE_HALOS` param + `:5817`; `barotropic_init` CS-member `map(to:)` block `:6579-6676` (list in §2.3 matches). +- `BT_cont_type` `MOM_variables.F90:317`; `alloc_BT_cont_type:567` maps `BT_cont` at `:582` then + member arrays right after their `allocate(...,source=0.0)`. +- Commit diffs `5f413739b` (ifort HYBRID re-fusion, 2 ins/10 del), `2108e0eba` (2 deletions, + nvfortran 25.5 / Stellar A100, `MOM_interface_heights.F90`), `b29b27150` (8 checksum + `target update from` insertions), `9f4e48be1` (added `omp_offload=.true.` to `pass_ubta_uhbta`), + `7003282d3` (added the deleted directive) — all verified line-for-line. +- `edoyango/acc-btstep`: 2 commits (`5bd8bb66d`, `1ff44b2c5`), `+289/−50`; 185 `!$acc kernels loop`, + 177 `async(`, 40 `!$acc wait`, 11 `omp_offload=.true.` (identical); 35 kernels + 12 waits inside + `btstep_timeloop`. All 10 small "port this loop" commit hashes in §2 exist with the quoted messages. + +**Corrected:** +- §1.2: wide-halo bounds set at `:6125-6126`, not `:6125-6127`. +- §2.2: the `:2070` row cross-referenced "§4", where `9f4e48be1` is not discussed; reworded to name the + commit's actual role (adding `omp_offload=.true.`). +- §5: the `async(1)` zero-init loop in the quoted snippet was truncated — it zeros an h-point *group* + (`gtot_*`, `eta`, `eta_PF`, and flag-gated `eta_PF_1/d_eta_PF/eta_IC/dyn_coef_eta`), not just the four + `gtot_*` arrays; snippet annotated and the miscount "four wide-halo zero-inits" corrected to three + loops (h/u/v groups), line range `:1018-1044`. + +**Enhancements:** added §6.5 (five sharpened transferable lessons — loop-fusion repro rule, +conditional-map-on-optional hazard, checksum-transfer tax, all-or-nothing wide-halo residency, the +`do concurrent`-for-compute / OpenMP-for-data split); added an "evidence discipline" paragraph to §5 +separating what the `acc-btstep` diff proves (additive, counts) from the unmeasured performance +inference. + +**Confidence:** High. Every line number, commit hash, diff, and directive count in the document was +checked against the tree and matched (modulo the three minor corrections above). The one genuinely +unverifiable claim — the OpenACC-async performance rationale — was already appropriately hedged in the +draft and is now flagged explicitly. diff --git a/knowledge/gpu-knowledge/10-inflight-ports.md b/knowledge/gpu-knowledge/10-inflight-ports.md new file mode 100644 index 0000000..8e5958c --- /dev/null +++ b/knowledge/gpu-knowledge/10-inflight-ports.md @@ -0,0 +1,559 @@ +# Naive vs. Blessed: A Contrast Study from In-Flight Branches + +> **Purpose.** `00-architecture.md` §0 defines the "blessed" port strategy (CPU-preserving +> k-blocking, mandatory bitwise reproducibility, `do concurrent` as the default idiom, no device +> polymorphism) in the abstract. This document makes it concrete by contrasting two unmerged +> branches that touch the *same* code (`MOM_density_integrals.F90` / EOS 3D interfaces / ALE PLM) +> but diverge on a *different* subsystem (Bodner MLE). Read `00-architecture.md` §0, §5, §6.2 first. +> +> Studied via `git log`/`git diff` only, no build. Branches: `bodner-naive-port` (forked from +> `dev/gpu` at `b40fe2d766` "add submodule support", **8 commits behind** the current `dev/gpu` tip +> `b8c471cfa` — it predates the merged EOS 2D/3D work `7c7af5572`, the block-repro-sum `8593a732a`, +> the k-block continuity `93dbbd36e`, the CorAdCalc k-block `b8c471cfa`, plus four earlier commits) +> and `port/pressureforce-benchmark_ALE` (forked at `c82e1254a6` "vertvisc: Fix CS memory +> management" — this **was** the `dev/gpu` tip when this study began, but `dev/gpu` has since +> advanced 4 commits past it, so it is now 4 behind). Where the two branches' *own* diffs disagree on +> `MOM_coms.F90` / `MOM_set_viscosity.F90` / `MOM_vert_friction.F90` / `MOM.F90`, it is **only +> baseline drift** (each forked at a different point on `dev/gpu`, and those files changed on +> `dev/gpu` in between). Verified the right way: **neither branch touches those files in its own +> work** — `git diff ..` on them is empty for `port/pressureforce-benchmark_ALE`, and +> `bodner-naive-port` has only a 3-line real edit to `MOM.F90` for its MLE port. (Do *not* verify +> this by diffing pf against the *current* `dev/gpu`: because `dev/gpu` advanced 4 commits — one of +> which, `8593a732a`, rewrote `MOM_coms.F90` — that diff is now large and misleading.) Do not read +> those files as branch content below. + +--- + +## 0. Branch topology — what is actually shared + +The commit graphs are not independent: both branches carry the **same ~24-commit PLM lineage** by +Edward Yang — from `submodulify` (`c75ebddaf`/`83628d990`), through "add j dimension" and the +`int_density_dz_generic_plm` tiling series, up to "move block size to user input" +(`b864bafba`/`1038a4921`) — that rewrites `int_density_dz_generic_plm` from a per-layer scalar-`k` +call into a k-blocked, submodule-split kernel. Verified by `git patch-id`: **21 of those commits are +byte-identical patches** across the two branches; the remaining 3 (`cec1c8d0a`/`8bbd4c085`/`207c0ce34` +— "use 2d calc dens" / "move k loops inside" / "block loop in set_pbvce") differ **only by EOS +baseline drift**: `bodner-naive-port` forked *before* the merged EOS 2D/3D commit, so on it these +three commits carry the `calculate_density_*_3d` additions inline (they each touch `MOM_EOS.F90` / +`MOM_EOS_Wright.F90` / `MOM_EOS_Roquet_rho.F90` / `MOM_EOS_base_type.F90`), whereas on +`port/pressureforce-benchmark_ALE` that machinery already arrived via its first commit `8eb41475b` +"EOS: 2D and 3D density implementations of methods". They then diverge at the **same timestamp** +(`Tue Jun 2 11:36:45 2026 -0400`, both authored by Edward Yang, both parented on the matched "move +block size to user input" commit) on one commit — `bodner-naive-port`'s +`8744bf362 set plm block size defaults to 32x4` vs. `port/pressureforce-benchmark_ALE`'s +`fae6c9a5c set plm block size defaults to 0x1` — a **literal fork point**, not two unrelated efforts. +(So "26 identical commits" is an overcount: the shared lineage is ~24 commits, 21 byte-identical and +3 differing only by the EOS drift above.) After the fork: + +- `bodner-naive-port` stops touching `MOM_density_integrals.F90`/PLM and instead layers the + **Bodner MLE (`MOM_mixed_layer_restrat.F90`) naive port** on top (`6f854c5aa` → `08be6d130`, + authored by "Jorge" with `Co-Authored-By: Claude Opus 4.8`). +- `port/pressureforce-benchmark_ALE` keeps refining the **same PLM density-integral kernel** + (`desubmodule` cleanup only). + +**Implication for judging "naive vs. blessed": the PLM/EOS k-blocking substrate is common, +in-flight, blessed-style work by a different author than the Bodner MLE port.** The genuinely +*naive* code under study is specifically `MOM_mixed_layer_restrat.F90` on `bodner-naive-port`. The +PLM/EOS work on both branches is judged separately below (§3) and is closer to blessed on both, +with `port/pressureforce-benchmark_ALE` slightly ahead. + +| | `bodner-naive-port` | `port/pressureforce-benchmark_ALE` | +|---|---|---| +| Forked from | `b40fe2d766` (**8** commits behind current `dev/gpu` `b8c471cfa`) | `c82e1254a6` (was tip at study time; now **4** behind) | +| Commits ahead of its fork | 30 | 27 | +| Files touched (own work) | `MOM_mixed_layer_restrat.F90` (**+174/−91**), `MOM_density_integrals.F90`/`_s.F90` (submodule split), EOS 3D interfaces (via the shared PLM lineage), `MOM_PressureForce_{FV,Montgomery}.F90` | `MOM_density_integrals.F90`, EOS 3D interfaces (deeper — Roquet_rho +158/−16), `MOM_PressureForce_{FV,Montgomery}.F90`, `MOM_ALE.F90`/`PLM_functions.F90` | +| Distinctive new file | `MOM_density_integrals_s.F90` (**submodule**, not a scalar variant — see §5) | none (same submodule pattern, later `desubmodule`d back) | + +--- + +## 1. The naive pattern, concretely — `MOM_mixed_layer_restrat.F90` on `bodner-naive-port` + +### 1.1 First cut (`6f854c5aa "naive omp offload working"`) + +Per-loop `!$omp target teams distribute parallel do collapse(2)` regions, each with its own +explicit `map(to/from/tofrom:)` clause, wrapped around loops copied verbatim from the CPU code. +Debug `print *` statements are left in (`print *, "Bodner"`, `print *, "MLD grid"`, `print *, "Else"`, +`print *, "anwer date"`, `print *, " if lfbod "`, … — **ten** newly-added debug prints, verified +against the fork baseline `b40fe2d766` which had only one pre-existing unit-test banner). They +survive through three further MLE commits and are stripped only by the last MLE commit +`08be6d130 "print removal"`. Note also the commit-message typos (`"denstiy"`, `"btiwse"`, +`"anwer date"`) consistent with fast, unreviewed iteration: + +```fortran +elseif (CS%use_Bodner) then + print *, "Bodner" + ! Implementation of Bodner et al., 2023 + call mixedlayer_restrat_Bodner(...) +``` + +```fortran +tau_bgrow = CS%BLD_growing_Tfilt ; tau_bdecay = CS%BLD_decaying_Tfilt +h_MLD_l(:,:) = h_MLD(:,:) +MLDf_l(:,:) = CS%MLD_filtered(:,:) +!$omp target teams distribute parallel do collapse(2) & +!$omp map(to: h_MLD_l) map(tofrom: MLDf_l) map(from: little_h) +do j=js-1,je+1 ; do i=is-1,ie+1 + little_h(i,j) = rmean2ts(h_MLD_l(i,j), MLDf_l(i,j), tau_bgrow, tau_bdecay, dt) + MLDf_l(i,j) = little_h(i,j) +enddo ; enddo +CS%MLD_filtered(:,:) = MLDf_l(:,:) +``` + +Two things to flag: (a) **`omp target teams distribute`** is used where the architecture doc says +`do concurrent` should be the default (principle #3) — this gets fixed two commits later; (b) a +**plain local copy `h_MLD_l`/`MLDf_l`** of every CS member is made before mapping "for clean device +mapping" (comment added later). This sidesteps CS-member/derived-type mapping headaches (§2.3 of +`00-architecture.md`) at the cost of doubling host memory and adding host-side copy loops that don't +exist in the CPU code path at all — a real, if small, CPU regression risk introduced by porting. +Density itself is still computed on the **host** in this commit: + +```fortran +! Active path: density at p=0 (sigma_0) computed on the HOST into rho3d (the polymorphic EOS +! dispatch stays on the host), then the per-column mixed-layer integral is offloaded. +do k=1,nz ; do j=js-1,je+1 + call calculate_density(tv%T(:,j,k), tv%S(:,j,k), p0, rho3d(:,j,k), tv%eqn_of_state, EOSdom) +enddo ; enddo +``` +followed by a `!$omp target teams distribute parallel do collapse(2)` per-column integral kernel +that maps `rho3d` in — i.e. a host EOS call feeding a device integral, a transfer the blessed +pattern (§1.2) tries to design away. + +### 1.2 Second cut (`02bf308f3 "persistent target data region + do concurrent for in-region loops"`) + +Wraps the *tail* of the routine (mixed-layer integral, U/V components, h update) in **one** +`!$omp target data` region and converts those loops to `do concurrent` + +`DO_LOCALITY(local(...))`: + +```fortran +!$omp target data & +!$omp map(to: little_h, big_H, wpup, Cr_l) map(alloc: rho3d) & +!$omp map(from: vol_dt_avail, htot, buoy_av, uhml, vhml, uDml_diag, vDml_diag) +... +do concurrent (j=js-1:je+1, i=is-1:ie+1) DO_LOCALITY(local(k, dh, Rml_i, htot_i)) + ... +enddo +... +!$omp end target data +``` + +This is a genuine step toward blessed (principle #1's "one source form" and principle #3's +`do concurrent` default), but it is applied only to the back third of the subroutine — the +earlier filter/w'u' loops (little_h, big_H, wpup) each keep their **own** `target enter +data`/`exit data` pair (confirmed: 5 `enter data`, 4 `exit data`, 2 `target data` blocks in the +final file), i.e. several small host↔device round-trips remain where the blessed pattern in +`MOM_continuity_PPM.F90`/`MOM_CoriolisAdv.F90` uses a single region spanning the whole hot path. + +### 1.3 Third cut (`09d1b93f7 "another do concurrent"`) + +Converts the remaining `omp target teams distribute` loops (little_h, big_H, wpup, wpup filter) to +`do concurrent`, each now bracketed by its own `target enter data ... exit data` (still not folded +into the one persistent region from §1.2). By branch tip, `omp target teams` no longer appears +anywhere in the file (0 occurrences) — principle #3 is fully satisfied in the end state, it just +took three iterations to get there, with the intermediate commits shipping a mixed idiom. + +**What is missing throughout, and never added:** `MOM_mixed_layer_restrat.F90` has **zero** +`niblock`/`njblock`/`nkblock` CS parameters anywhere (checked: no `block` hits besides unrelated +`openParameterBlock` calls). There is no CPU-cache-blocking story at all — every device loop runs +over the whole horizontal domain unconditionally, with no `#ifdef __NVCOMPILER_OPENMP_GPU` / +CPU-default split like the one added to `MOM_PressureForce_FV.F90` (§3). Compare to principle #1: +the blessed pattern requires *one source form that is also tuned for CPU*; this port has one source +form that was **never tuned for CPU at all** — the whole-domain assumption is baked in with no +runtime override, unlike every merged/blessed kernel in `00-architecture.md` §5/§6.1. + +--- + +## 2. "Call calculate density on the GPU bitwise" (`2271af66e`) — how it stays bit-identical + +Before this commit, the p=0 (sigma_0) density used for the Bodner mixed-layer integral was computed +by a **host** loop calling the existing 1-D `calculate_density` interface column-by-column, then +`!$omp target update to(rho3d)` pushed the result to device. `2271af66e` replaces that with a +single **3-D** EOS call issued from inside the target-data region: + +```fortran +T_l(:,:,:) = tv%T(:,:,:) ; S_l(:,:,:) = tv%S(:,:,:) +p3d(:,:,:) = 0.0 +EOSdom3d(1,:) = EOS_domain(G%HI, halo=1) +EOSdom3d(2,:) = [(js-1) - (G%jsd-1), (je+1) - (G%jsd-1)] +EOSdom3d(3,:) = [1, nz] +... +! Active path: density at p=0 (sigma_0) computed ON THE DEVICE via the 3D EOS interface (the +! polymorphic dispatch is resolved host-side, the per-element evaluation runs in do concurrent), +! then the per-column mixed-layer integral is offloaded. +call calculate_density(T_l, S_l, p3d, rho3d, tv%eqn_of_state, EOSdom3d) +``` + +This only compiles/works because a **`calculate_density_3d` generic** and matching 3-D type-bound +procedures already exist in the EOS layer of this branch. **They were *not* added by `2271af66e` +itself** — that commit only edits `MOM_mixed_layer_restrat.F90` (+17/−8) to *call* the 3-D +interface. The 3-D EOS machinery arrived earlier, inside the shared Edward-Yang PLM lineage +(`cec1c8d0a`/`8bbd4c085`/`207c0ce34` — the same three commits flagged as EOS baseline drift in §0). +Because `bodner-naive-port` forked *before* the merged EOS 2D/3D commit `7c7af5572`, the 3-D +interface on this branch is carried by that PLM lineage rather than inherited from the merge; it +mirrors the same 2D pattern that `7c7af5572` merged onto `dev/gpu`: + +- `MOM_EOS.F90`: `calculate_density_3d` added to the `calculate_density` interface; it resolves + scaling (`EOS%RL2_T2_to_Pa` etc.) then calls `EOS%type%calculate_density_array_3d(...)` — the + **one place** the polymorphic v-table dispatch happens, on the host, before any device region is + entered. +- `MOM_EOS_base_type.F90`: default fallback `a_calculate_density_array_3d` — still polymorphic + (`class(EOS_base), intent(in) :: this`), calls the elemental `this%density_elem(...)` in + whole-array syntax. **This fallback is not GPU-safe** — only concrete EOS classes with their own + override are. +- `MOM_EOS_Wright.F90` / `MOM_EOS_Roquet_rho.F90`: concrete overrides + `calculate_density_array_3d_buggy_Wright` / `..._Roquet_rho` implement the device-safe path with + `do concurrent` over a free `_loc` function (no `this`): + +```fortran +! NOTE: There is an implicit copy of `this` which cannot yet be prevented. +! Possibly because Nvidia cannot associate `this` with `EOS%type`. +if (present(rho_ref)) then + do concurrent (k=ks:ke, j=js:je, i=is:ie) + rho(i,j,k) = density_anomaly_elem_buggy_Wright(this, T(i,j,k), S(i,j,k), pressure(i,j,k), rho_ref) + enddo +else + do concurrent (k=ks:ke, j=js:je, i=is:ie) + rho(i,j,k) = density_elem_buggy_Wright_loc( T(i,j,k), S(i,j,k), pressure(i,j,k)) + enddo +endif +``` + +**Why this is bitwise-safe:** the `rho_ref`-absent branch calls exactly the same per-element free +function (`density_elem_buggy_Wright_loc`) that the already-merged 2D path calls — same polynomial, +same operation order, only the surrounding loop nest and *where* (host vs. device) it runs have +changed. No reduction, no reordering, no fused-multiply-add reassociation. The commit re-uses the +architecture's established "resolve the v-table once on the host, then call a free `_loc` kernel +inside the parallel region" idiom from `07-eos` and matches the "implicit copy of `this`" nvfortran +limitation already catalogued in `00-architecture.md` §7.5 — the `present(rho_ref)` branch above +*still* passes `this` and is annotated with that exact caveat, i.e. it is a **known, not-yet-fixed** +gap even in this "GPU bitwise" commit: the rho_ref path is not actually proven device-safe, only the +no-rho_ref path (the one Bodner MLE actually uses) is. + +--- + +## 3. `port/pressureforce-benchmark_ALE`'s k-blocking of `int_density_dz_generic_plm` + +This is the blessed template — CS-level block-size parameters with CPU/GPU-conditional defaults, +resolved at `_init`, `0` meaning "whole domain": + +```fortran +#ifdef __NVCOMPILER_OPENMP_GPU +integer, parameter :: default_nkblock = 0 !< 0 = full domain +integer, parameter :: default_njblock_plm = 0 +#else +integer, parameter :: default_nkblock = 1 !< 1 = one layer per CPU cache-blocking pass +integer, parameter :: default_njblock_plm = 1 +#endif +integer, parameter :: default_niblock_plm = 0 !< i is never cache-blocked here +``` +(`fae6c9a5c "set plm block size defaults to 0x1"` — the "0x1" in the commit title is +`niblock=0 (full), njblock=1 (one row)`, with `nkblock` fixed at 1 on CPU / 0 on GPU by the earlier +`#ifdef` block. Contrast `bodner-naive-port`'s stalled `8744bf362 "set plm block size defaults to +32x4"`, i.e. `niblock=32, njblock=4` — a direct copy of the `MOM_continuity_PPM.F90` "32/4/1" +convention cited in `00-architecture.md` §5, before further benchmarking on this branch revised it +down to row/full/layer tiling more suited to the EOS-heavy PLM kernel.) + +The interface gains `niblock`/`njblock` optional arguments (`1038a4921`/`b864bafba "move block size +to user input"`, identical on both branches) that are threaded down into +`generic_plm_update_{dpa,intx_dpa,inty_dpa}`'s `TILE_SIZE_X`/`TILE_SIZE_Y` locals. + +**"move k loops inside" (`3b8525a54` on pf / `8bbd4c085` on bodner — the *same* logical step on both +branches; the two differ only by the EOS baseline drift noted in §0, since this commit also touches +`MOM_EOS*.F90`)** is the substantive k-blocking step: scratch arrays (`T5`,`S5`,`p5`,`r5`,`u5`, …) grow a third `kstart:kend` +dimension, and the per-`k` `calculate_density` call over a 2-D `(5*i,j)` domain is replaced by +**one** `calculate_density` call over the whole 3-D `(5*i,j,k)` block: + +```fortran +real :: T5(5*TILE_SIZE_X,TILE_SIZE_Y,kstart:kend) ! was (5*TILE_SIZE_X,TILE_SIZE_Y) +... +EOSdom_h5(3,1) = 1 ; EOSdom_h5(3,2) = kend-kstart+1 +if (use_rho_ref) then + call calculate_density(T5, S5, p5, r5, EOS, EOSdom_h5, rho_ref=rho_ref) ! one 3D call, not a k-loop of 2D calls +else + call calculate_density(T5, S5, p5, r5, EOS, EOSdom_h5) + do concurrent (k=kstart:kend, j=jstart:jend, i=istart:iend, n=1:5) + ... + enddo +endif +``` +i.e. exactly the amortize-the-EOS-dispatch-over-k pattern that `00-architecture.md` §7.1 identifies +as the point of the 3D EOS interface, applied here to the PGF-integral hot path rather than to the +Bodner MLE code. + +**"optimise a little bit" (`2a99c9dd1` on pf = `98cb748c6` on `bodner-naive-port` — patch-id +identical, so present on *both* branches)** applies the vertvisc-style tridiagonal idiom (§4.3/§5 of +`00-architecture.md`: outer parallel loop, serial recurrence loop, inner parallel loop) to the three +places in `PressureForce_FV_Bouss` that have a genuine `k`-recurrence (the interface-height `e` and +the `intx_pa`/`inty_pa` cumulative sums) and previously used an outer-serial-`k` / +inner-`do-concurrent` nesting the wrong way round: + +```fortran +! before: do k=nz,1,-1 ; do concurrent (j=...,i=...) ... enddo ; enddo +do concurrent(j=Jsq:Jeq+1) + do k=nz,1,-1 ! true recurrence in k stays serial + do concurrent (i=Isq:Ieq+1) + e(i,j,K) = e(i,j,K+1) + h(i,j,k)*GV%H_to_Z + enddo + enddo +enddo +``` +This is a genuine "did we get the k-blocking direction right" fix. The commit's *entire* content is: +the three recurrence-direction reversals above (25 changed lines in `MOM_PressureForce_FV.F90`), a +`private(k)` fix on one `!$omp target teams loop`, one added +`!$omp target enter data map(to: tv_tmp, tv_tmp%T, tv_tmp%S)`, and a 1-line change to `MOM_ALE.F90`. +**Because `98cb748c6` and `2a99c9dd1` are byte-identical, `bodner-naive-port` carries this same fix** +— it is *not* a pf-vs-bodner differentiator. + +> **CORRECTION (verified).** An earlier draft attributed to this commit the removal of a +> `! defensive update - not sure if it works` directive and a narrowing of an +> `!$omp target update from(e)` guard from `Recon_Scheme > 0` to `Recon_Scheme == 2`. **Neither is +> real.** The string `defensive`/`not sure if it works` appears *nowhere* in either branch's `src/`, +> and the `!$omp target update from(e) if(...)` guard is the identical `Recon_Scheme == 2` form on +> both branches. Directly diffing the two branches' `MOM_PressureForce_FV.F90` yields **only** the +> 9-line block-size-defaults hunk (`32x4` vs `0x1`) — no transfer-guard or defensive-directive +> divergence exists. + +**Is this the blessed template applied to pressure integrals? Yes**, modulo one caveat: unlike +`MOM_continuity_PPM.F90`'s manual `nteams` team-count tuning (`00-architecture.md` §5, working +around commit `5b5f6b2b1`'s under-launch bug), this branch relies entirely on `do concurrent` — no +`!$omp target teams num_teams(...)` anywhere in the diff — so if nvfortran under-launches teams +for the 5×/15×-widened `T5`/`T15`/etc. arrays the way it did for continuity, that workaround has not +yet been ported over here. Flag for whoever picks this up for merge. + +> **Open (reviewed 2026-07-14):** does the PLM density-integral hot path actually need continuity's +> manual `num_teams(ceiling(...))` workaround, or does the tile geometry here (a `5*TILE_SIZE_X` inner +> dimension) keep nvfortran's default team launch adequate? The review could not settle this from +> source — it needs a benchmark of the PLM kernel against the under-launch symptom that motivated +> `5b5f6b2b1` in continuity. See KNOWLEDGE.md §9. + +--- + +## 4. Same code, two branches — what diverges + +| Aspect | `bodner-naive-port` (PLM/EOS portion) | `port/pressureforce-benchmark_ALE` | +|---|---|---| +| PLM k-blocking depth | **Identical.** Carries the full lineage incl. "move k loops inside" (EOS amortized over the whole k-block) and "optimise a little bit" (recurrence-direction fix), `98cb748c6` = pf's `2a99c9dd1` byte-for-byte | Same lineage; diverges from bodner *only* at the block-size-defaults commit and the later `desubmodule` | +| Block-size defaults (CPU side of the `#ifdef`; GPU side is `0/0/0` on both) | `32x4` (`niblock=32, njblock=4`) | `0x1` (`niblock=0, njblock=1`) — revised after benchmarking this specific EOS-heavy kernel | +| Submodule split | Kept (`_s.F90` present at tip) | Reversed (`a3e889601 desubmodule`, `_s.F90` deleted) | +| Roquet_rho diff-from-fork | +77 | +158/−16 (deeper — but its diff-from-fork *also* absorbs the `8eb41475b` EOS-merge equivalent that bodner instead folds into the PLM-lineage commits) | +| **Closer to merge-quality on this shared file** | — | **Marginally.** The only substantive edges are the benchmarked-down block defaults and the `desubmodule` cleanup; the k-blocking body (incl. the recurrence-direction fix) is identical on both. | + +> **Open (reviewed 2026-07-14):** is `port/pressureforce-benchmark_ALE` genuinely the more merge-ready +> branch on the shared PLM code, or merely *different*? Its only edges are the `0x1` CPU default and +> the `desubmodule`. The review could not settle this from source — it needs a benchmark (does `0x1` +> beat `32x4` on CPU?) and a maintainer decision on whether desubmoduling is the intended end-state. +> See KNOWLEDGE.md §9. + +The one thing `bodner-naive-port` has that the other branch doesn't touch at all is the actual +**Bodner MLE port** — but that work, per §1, is the naive contrast case, not a competing +implementation of the same feature. + +--- + +## 5. `MOM_density_integrals_s.F90` — what the `_s` suffix actually means + +**It is not a scalar/structured duplicate and not a device-hazard workaround.** `git show +bodner-naive-port:src/core/MOM_density_integrals_s.F90 | head` shows: + +```fortran +!> Provides integrals of density +submodule (MOM_density_integrals) MOM_density_integrals_s +``` + +`_s` = **submodule**. The `submodulify`/`desubmodule` commit pair (`c75ebddaf` on `bodner-naive-port`, +`83628d990`/`a3e889601` on `port/pressureforce-benchmark_ALE`) is a Fortran `module`/`submodule` +split: `MOM_density_integrals.F90` shrinks to public declarations + an `interface … module subroutine +… end interface` block, and the executable bodies move verbatim into +`MOM_density_integrals_s.F90` (`submodule (MOM_density_integrals) MOM_density_integrals_s`, +`module subroutine int_density_dz_generic_plm(...)` matching the interface). The +`git log --graph` history shows a **sibling, unrelated branch** doing the identical maneuver +project-wide (`"+Convert all modules to module+submodule pairs for compile-speed testing"`) — +this is a **build/compile-time separation technique** (submodules let the interface-only file be +a stable compilation unit that downstream users don't need to recompile when the implementation +changes), not a GPU-porting device-hazard fix. `port/pressureforce-benchmark_ALE` in fact +**`desubmodule`s it back** (`a3e889601`) at the very end of its own history, i.e. the split was +provisional/exploratory scaffolding on both branches, later abandoned on the more mature branch. +Net effect on the diff stat (`MOM_density_integrals.F90` −518 / `MOM_density_integrals_s.F90` +589 +on `bodner-naive-port`) is overwhelmingly code *moved*, not duplicated. + +--- + +## 6. Lessons — what a naive port gets wrong that the blessed pattern fixes + +1. **No CPU-blocking story at all vs. a tuned one.** `MOM_mixed_layer_restrat.F90` ships with zero + `niblock`/`njblock`/`nkblock` parameters — the device version *is* the only version, so there is + no way to reason about (or preserve) CPU cache performance. The PLM work on both branches, by + contrast, exposes `PGF_PLM_NKBLOCK`/`NIBLOCK`/`NJBLOCK` runtime params with `#ifdef + __NVCOMPILER_OPENMP_GPU`-gated defaults from day one, and iterates the CPU-side default + (`32x4` → `0x1`) as understanding improves. **Lesson: add the block-size CS parameters and the + `#ifdef` default split in the *first* commit, not as an afterthought — it is what makes "one + source form for CPU and GPU" (principle #1) achievable instead of aspirational.** + +2. **`omp target teams distribute` before `do concurrent`.** The naive branch's first commit reaches + for classic OpenMP target constructs; two commits later it's rewritten to `do concurrent` + + `DO_LOCALITY`. Both compile and (per the branch's own commit message) are "bitwise", but the + churn shows the default idiom (principle #3) wasn't the first instinct — a porting agent + following the doc should reach for `do concurrent` immediately and reserve `omp target teams` + for the reduction/underperformance cases the architecture doc names explicitly. + +3. **Many small `target enter/exit data` pairs vs. one persistent region.** Even at branch tip, + `MOM_mixed_layer_restrat.F90` has 5 `enter data` / 4 `exit data` sites plus 2 `target data` + blocks — most of the routine still round-trips host↔device multiple times per call. Only the + integral/U/V/h-update tail was folded into a single persistent region. The blessed exemplars + (`MOM_continuity_PPM.F90`, `MOM_CoriolisAdv.F90`) hoist communication/mapping to the driver and + keep one region per hot path. **Lesson: design the data-region boundary before writing loops, + not loop-by-loop.** + +4. **Host EOS calls feeding device integrals, fixed by extending the interface, not duplicating + logic.** The first naive commit computes `rho3d` on the host and `target update to`s it in — an + extra round trip and, worse, a hidden serialization point between "compute density" and + "consume density" that the second (`2271af66e`) commit removes by switching the MLE code to a + genuine 3-D `calculate_density` entry point (the EOS-layer machinery for which the shared PLM + lineage had already added — mirroring the merged 2D pattern) rather than hand-inlining the + Wright/Roquet math into `MOM_mixed_layer_restrat.F90`. This preserved + bitwise results because the same `_loc` free-function kernel is reused; it would not have if the + formula had been retyped locally. + +5. **Polymorphism is still not fully closed.** Even in the "GPU bitwise" commit, the + `present(rho_ref)`-true branch of `calculate_density_array_3d_buggy_Wright` still passes `this` + into a `do concurrent` loop with an explicit unresolved-limitation comment. The base-type + fallback `a_calculate_density_array_3d` is polymorphic outright and silently unsafe for any EOS + form that doesn't override it (per `00-architecture.md` §7.1, most forms still don't). A porting + agent should treat "we added a 3D interface" as necessary but not sufficient — check whether the + *specific* branch taken at runtime (which `rho_ref`/EOS-form combination) actually reaches a + `_loc`-based override before calling it device-safe. + +6. **Debug artifacts linger without review discipline.** Ten `print *` debug statements survived + through three further MLE commits before `08be6d130 "print removal"` stripped them, and several + commit messages carry typos (`"denstiy"`, `"btiwse"`, `"anwer date"`) — the signature of fast, + unreviewed iteration. (Note: contrary to an earlier draft, the naive branch does *not* carry a + leftover `! defensive update` directive or a coarser transfer guard than pf — its + `MOM_PressureForce_FV.F90` differs from pf's by exactly the 9-line block-size-defaults hunk and + nothing else; the "CPU-tuning is a distinct phase" point is real but lives in the `32x4 → 0x1` + default change of Lesson 1, not in any transfer-directive cleanup.) **Lesson: strip debug prints + and fix message typos before proposing merge — they are the cheapest possible signal that a port + has not been reviewed.** + +--- + +## Naive-vs-blessed checklist (derived from this comparison) + +| Trait | Naive (`bodner-naive-port` MLE code) | Blessed (PLM/EOS work, both branches; merged exemplars in §5/§6.1 of `00-architecture.md`) | +|---|---|---| +| Default parallel idiom | Starts `omp target teams distribute`, migrates to `do concurrent` over 3 commits | `do concurrent` (+ `DO_LOCALITY`) from the start; `omp target teams` reserved for reductions/underperformance | +| CPU block-size parameters | None | `niblock`/`njblock`/`nkblock` CS params, `#ifdef`-gated defaults, tuned by benchmarking | +| Data-region granularity | Many small per-loop `enter/exit data` + one late persistent region for the tail only | One region per hot path/subroutine call | +| EOS density calls | First commit: host loop + `target update to`; fixed only in a follow-up commit | 3-D interface added at the EOS layer, dispatched once (host, v-table) then `do concurrent` + `_loc` kernel | +| Polymorphism (`this`) | N/A here (fixed via the EOS-layer fix, but the fix itself still has one unresolved `this`-copy branch) | Documented, tracked, not fully eliminated even on the "blessed" branch — an open item, not a solved one | +| Cleanliness | Ten `print *` debug statements committed and removed only in the last commit; typos in commit messages (`"denstiy"`, `"btiwse"`) | No debug prints; k-recurrences nested correctly (recurrence-direction fix — though that fix is *shared* with the naive branch, not exclusive to the blessed one) | +| Bitwise care | Claimed ("Bitwise" in commit message) and plausible by construction (same `_loc` formulas, relocated loops) — not independently checksum-verified in this study | Same standard; also not independently checksum-verified here (source+git only, per constraints) | + +--- + +## Review rubric — run against your own diff *before* proposing a merge + +Eight pass/fail gates distilled from the contrast above. Each cites the concrete evidence a reviewer +can re-check, so "why does this matter?" always has an answer in this repo's history. + +1. **Block-size CS parameters present and `#ifdef`-gated?** + `grep` your diff: every hot-loop file must add `ni/nj/nkblock`-style CS integers, `get_param`'d + (e.g. `PGF_PLM_NKBLOCK` / `_NIBLOCK` / `_NJBLOCK`, `MOM_PressureForce_FV.F90:2297-2308`), behind an + `#ifdef __NVCOMPILER_OPENMP_GPU` default split — GPU `0` (whole domain), CPU a real tile size + (`MOM_PressureForce_FV.F90:41-48`). **Fail:** `MOM_mixed_layer_restrat.F90` on `bodner-naive-port` + — *zero* block params (verified: no `niblock`/`njblock`/`nkblock` hits), whole-domain assumption + baked in with no CPU override. → Add these in the **first** commit, not as an afterthought. + +2. **One persistent data region per hot path — not many small `enter/exit data` pairs?** + Count `enter data` / `exit data` / `target data` per touched file; they should collapse toward a + single region spanning the call. **Fail:** MLE tip still has **5 `enter data` / 4 `exit data` / 2 + `target data`** (verified) — most of the routine round-trips host↔device per call. **Pass:** the + continuity/CorAdCalc drivers hoist mapping up and keep one region per hot path. + +3. **`do concurrent` (+`DO_LOCALITY`) the default idiom — `omp target teams` only for + reductions/underperformance?** + `grep 'target teams distribute'`; there should be none left except documented reduction/column + cases. **Fail signal from history:** MLE first cut shipped **9** `target teams distribute` and 0 + `do concurrent`; it took three commits to reach 0/10 (verified counts). Reach for `do concurrent` + immediately. + +4. **EOS density via the extended 2D/3D interface, v-table resolved once host-side?** + No host `calculate_density` loop feeding a `!$omp target update to`. The call must be a single + 2D/3D `calculate_density(...)` that resolves scaling on the host (`EOS%RL2_T2_to_Pa …`, + `MOM_EOS.F90`) then runs `do concurrent` over a free `_loc` kernel (no `this`). **Evidence:** + `2271af66e` (MLE) and the "move k loops inside" PLM commit both do this; the naive first cut did + the host-loop→`target update to` anti-pattern. + +5. **Bitwise safety — argue it, and check *which* runtime path you hit.** + For every relocated loop confirm the per-element arithmetic is byte-identical: same `_loc` + function, same operand order, no new reduction / FMA reassociation. **Caveat to check + explicitly:** the `present(rho_ref)` branch of `calculate_density_array_3d_buggy_Wright` *still* + passes polymorphic `this` (verbatim comment: "implicit copy of `this` … cannot yet be prevented"). + Verify your runtime `rho_ref`/EOS-form combination reaches a `_loc`-based override, **not** the + polymorphic base-type fallback `a_calculate_density_array_3d` (unsafe for any EOS form lacking an + override — per `00-architecture.md` §7.1 most forms still lack one). + +6. **No debug prints, no commit-message typos going into the merge?** + `grep 'print \*'`; strip them. **Evidence:** ten debug prints survived three MLE commits before + removal — the cheapest possible signal that a diff has not been reviewed. + +7. **CPU-tuning phase actually done — not just "compiles + bitwise once"?** + Block-size CPU defaults must be *benchmarked*, not copy-pasted. **Evidence:** the CPU default was + iterated `32x4 → 0x1` on the PLM kernel because the `MOM_continuity_PPM.F90` `32/4/1` convention + did not suit the EOS-heavy PLM loop. A diff whose CPU default is a verbatim copy of another + kernel's is a red flag until benchmarked. + +8. **`k`-recurrences nested the right way?** + Any genuine `k`-recurrence (cumulative sums, `e(:,:,K)=e(:,:,K+1)+…`) must be + `do concurrent(j) → serial do k → do concurrent(i)`, never a `serial do k` wrapping a full-2D + `do concurrent(j,i)`. **Evidence:** the "optimise a little bit" commit (present on *both* branches) + reverses exactly this in `PressureForce_FV_Bouss` for `e`, `intx_pa`, and `inty_pa`. + +--- + +## Verification notes + +Verified by an Opus agent against `git` (branches `bodner-naive-port`, `port/pressureforce-benchmark_ALE`, +`dev/gpu`) and source only; no build/run. Temp artifacts under `tmp_local_artifacts/` (none retained). + +**Confirmed (spot-checked against source/git):** +- Same-timestamp literal fork point: `8744bf362` (32x4) and `fae6c9a5c` (0x1), both + `Tue Jun 2 11:36:45 2026 -0400`, both by Edward Yang, both parented on the patch-id-matched "move + block size to user input" pair (`b864bafba`/`1038a4921`). +- MLE idiom migration counts: `target teams distribute` 9→4→0→0→0; `do concurrent` 0→5→9→10→10 across + `6f854c5aa`→`02bf308f3`→`09d1b93f7`→`2271af66e`→`08be6d130`. Tip: 0 `omp target teams`, 5 `enter + data`, 4 `exit data`, 2 `target data`, **zero** block-size params. All verbatim. +- §2 EOS code (host `RL2_T2_to_Pa` dispatch, base-type polymorphic fallback + `a_calculate_density_array_3d`, Wright override with `rho_ref`→`this` / else→`_loc`, and the + "implicit copy of `this` … cannot yet be prevented" comment) — all present verbatim on bodner tip. +- §5 submodule facts: `submodule (MOM_density_integrals) MOM_density_integrals_s` (line 8); + `port/pressureforce-benchmark_ALE` deletes `_s.F90` via `a3e889601 desubmodule`; sibling branch + `ecbf83a8d "+Convert all modules to module+submodule pairs for compile-speed testing"` exists; + cumulative bodner diff `−518 / +589`. +- §3 `#ifdef __NVCOMPILER_OPENMP_GPU` block, `PGF_PLM_*` params, the recurrence-direction code, the + T5 `(…,kstart:kend)` 3-D scratch arrays + single 3-D `calculate_density`, and the §1.2 + `target data … DO_LOCALITY(local(...))` region — all present as quoted (§3 quoted call signatures + are lightly idealized paraphrases, but the transformation they depict is real). +- MLE commits authored by "Jorge" with `Co-Authored-By: Claude Opus 4.8 `. + +**Corrected:** +1. Fork positions: bodner is **8** commits behind current `dev/gpu` tip `b8c471cfa` (not 3); pf's fork + `c82e1254a6` is **no longer** the tip (now 4 behind). The "pf diff vs `dev/gpu` is empty" drift + test is stale — the correct test is pf's diff vs its *own* fork (empty), and bodner has a 3-line + real `MOM.F90` edit. +2. Shared lineage is **~24 commits (21 byte-identical by `git patch-id` + 3 EOS-drift)**, not "26 + identical". The 3 drifting commits (`cec1c8d0a`/`8bbd4c085`/`207c0ce34`) carry the EOS 3-D + additions inline on bodner because it forked before the merged EOS commit. +3. **The 3-D EOS interface was not added by `2271af66e`** (which only edits MLE, +17/−8, to *call* + it) — it came from the shared PLM lineage. +4. **Fabricated evidence removed:** no `! defensive update - not sure if it works` directive exists + in either branch, and there is no `Recon_Scheme > 0 → == 2` transfer-guard narrowing — the two + branches' `MOM_PressureForce_FV.F90` differ by *only* the 9-line block-size-defaults hunk. +5. **§4 "PLM k-blocking depth" divergence was false:** bodner carries the *full* lineage including + the "optimise a little bit" recurrence-direction fix (`98cb748c6` = pf's `2a99c9dd1`, patch-id + identical). Real post-fork divergence = block-size defaults (`32x4` vs `0x1`) + pf's `desubmodule`. +6. Numbers: MLE own-work stat `+174/−91` (not `+265`); newly-added debug prints = **ten** (not five); + Roquet_rho pf diff `+158/−16`. + +**Confidence:** High on all branch-forensics (patch-id, timestamps, file diffs re-derived +independently). High on the §2/§3/§5 code-content confirmations (read on the branch blobs). The open +items in §3 and §4 are genuinely benchmark-dependent judgments (team-launch adequacy; whether pf +is materially more merge-ready) that a source-only study cannot settle. diff --git a/knowledge/gpu-knowledge/11-halos-domains-multigpu.md b/knowledge/gpu-knowledge/11-halos-domains-multigpu.md new file mode 100644 index 0000000..bb16156 --- /dev/null +++ b/knowledge/gpu-knowledge/11-halos-domains-multigpu.md @@ -0,0 +1,756 @@ +# Halos, Domain Decomposition, and Multi-GPU Communication (dev/gpu) + +> **Purpose.** How MOM6's domain decomposition and halo-exchange infrastructure +> (`MOM_domains.F90` → `config_src/infra/FMS2/MOM_domain_infra.F90` → FMS `mpp_domains`) was adapted +> for GPU residency, and what breaks when the domain is spread across more than one GPU. Companion to +> `00-architecture.md` §3.3 (index/halo conventions) and §7.3 (halo/`omp_offload` summary), and to +> `09-barotropic-solver.md` (deep dive on the wide-halo `CS%BT_Domain` sub-cycle — this document only +> summarizes that mechanism and instead concentrates on the cross-module `omp_offload` inventory, the +> nonblocking pattern, and the multi-GPU tracer-advection bugfix). All line numbers are against +> `dev/gpu` unless a commit hash is given. + +--- + +## 1. Domain decomposition and the halo model + +### 1.1 Data domain vs. computational domain + +Every PE (MPI rank) owns a rectangular tile of the global grid. `hor_index_type` +(`src/framework/MOM_hor_index.F90:18`), replicated into `ocean_grid_type` (`src/core/MOM_grid.F90:28`), +carries two sets of bounds per staggering: + +- **Computational domain** — `isc:iec`, `jsc:jec` — the cells this PE actually owns and updates. +- **Data domain** — `isd:ied`, `jsd:jed` — computational domain **+ halo** — the cells this PE can + read, filled by neighbor exchange. +- **Global domain** — `isg:ieg` — indices into the whole simulation, used only for I/O/diagnostics. + +`NIHALO_ = NJHALO_ = 2` is the default halo width (`00-architecture.md` §3.1, `config_src/memory/`). +In **symmetric memory** mode, the B/C-grid (velocity/corner) index `IsdB` starts one lower than `isd` +(`MOM_hor_index.F90:90-96`: `HI%IsdB = HI%isd ; ... ; if (HI%symmetric) HI%IsdB = HI%isd - 1`), so a +`u`-point array is `u(IsdB:IedB, jsd:jed)` and a corner array is `q(IsdB:IedB, JsdB:JedB)`. This is +purely an indexing convention (`MOM_memory_macros.h` `NIMEMB_`/`NIMEMB_SYM_`); it does not change how +many halo rings are physically exchanged. + +A `MOM_domain_type` (`config_src/infra/FMS2/MOM_domain_infra.F90`) wraps an FMS `domain2D` +(`mpp_domain` member) plus MOM-specific bookkeeping (`nihalo`/`njhalo`, `symmetric`, +`nonblocking_updates`, `thin_halo_updates`). `G%Domain` is the "normal" halo=2 domain used by nearly +everything; `CS%BT_Domain` in `MOM_barotropic.F90` is a **second, wider-halo** domain cloned from it +(§5). + +### 1.2 `create_group_pass` — batching multiple fields into one exchange + +`MOM_domains.F90:51` re-exports the whole halo-update API from +`config_src/infra/FMS2/MOM_domain_infra.F90`: + +```fortran +public :: create_group_pass, do_group_pass, group_pass_type, start_group_pass, complete_group_pass +``` + +`create_group_pass` is a generic interface (`MOM_domain_infra.F90:91-96`) over 2D/3D scalar and vector +field registration (`create_var_group_pass_2d/3d`, `create_vector_group_pass_2d/3d`). The batching +trick: callers pass the **same** `group_pass_type` variable (a CS member, e.g. `CS%pass_h`, +`CS%pass_uv`) across several calls, each with a different field: + +```fortran +! src/core/MOM_dynamics_split_RK2.F90:512-514 +call create_group_pass(CS%pass_hp_uv, hp, G%Domain, halo=cor_stencil) +call create_group_pass(CS%pass_hp_uv, u_av, v_av, G%Domain, halo=max(cor_stencil,vel_stencil)) +call create_group_pass(CS%pass_hp_uv, uh(:,:,:), vh(:,:,:), G%Domain, halo=max(cor_stencil,vel_stencil)) +``` + +Internally (`create_var_group_pass_3d`, `MOM_domain_infra.F90:985-1026`): + +```fortran +if (mpp_group_update_initialized(group)) then + call mpp_reset_group_update_field(group, array) ! append another field to an existing group +elseif (present(halo) .and. MOM_dom%thin_halo_updates) then + call mpp_create_group_update(group, array, MOM_dom%mpp_domain, flags=dirflag, position=position, & + whalo=halo, ehalo=halo, shalo=halo, nhalo=halo) +else + call mpp_create_group_update(group, array, MOM_dom%mpp_domain, flags=dirflag, position=position) +endif +``` + +The first call on a given `group_pass_type` initializes it (`mpp_create_group_update`); every +subsequent call with a *different array* on the *same* `group` variable is detected as +"already initialized" and appends the field via `mpp_reset_group_update_field` instead of +re-creating the group. One `do_group_pass(group, ...)` later then issues **a single underlying +MPI exchange batching all registered fields** (u, v, h, T, S, ... in one message per neighbor +direction), instead of one exchange per field. This is a pre-existing FMS/MOM6 optimization (predates +the GPU port) that the GPU port reuses unchanged — the `create_group_pass` calls in +`MOM_dynamics_split_RK2.F90:506-517` are re-executed every dycore call (cheap: they only touch the +group's field list, not the data), and `do_group_pass` is what actually moves halo data. + +Per-pass halo width is tailored to the *consuming* kernel's stencil rather than always using the +maximum `NIHALO_=2`: `cor_stencil = CoriolisAdv_stencil(...)`, `vel_stencil = max(2, obc_stencil, +hor_visc_vel_stencil(...))`, `cont_stencil = continuity_stencil(...)` +(`MOM_dynamics_split_RK2.F90:498-504`), and each `create_group_pass` call passes `halo=` a `max(...)` +of whichever stencils touch that particular field. `pass_eta` uses `halo=1` (`:506`) because only a +1-point stencil ever reads `eta`'s halo. Smaller halo ⇒ smaller message ⇒ less data crossing the +device/host and PE/PE boundary per exchange — directly relevant to GPU communication cost. + +--- + +## 2. The `omp_offload` mechanism — GPU-aware halo exchange + +### 2.1 The infra change + +Baseline (`dev-gfdl`) `do_group_pass` took no offload argument. Commit `656e09013` ("add flag for +gpu2gpu do_group_update mpi transfers for latest fms", the only commit touching +`config_src/infra/FMS2/MOM_domain_infra.F90` between `dev-gfdl` and `dev/gpu` — +`git log --oneline dev-gfdl..dev/gpu -- config_src/infra/FMS2/MOM_domain_infra.F90` shows only a +license-relicense commit and a `dev/gfdl` merge besides it) added the parameter: + +```fortran +! config_src/infra/FMS2/MOM_domain_infra.F90:1143-1162 +subroutine do_group_pass(group, MOM_dom, clock, omp_offload) + type(group_pass_type), intent(inout) :: group + type(MOM_domain_type), intent(inout) :: MOM_dom + integer, optional, intent(in) :: clock + logical, optional, intent(in) :: omp_offload !< Whether the data to be transferred is + !! offloaded to the GPU with OpenMP. + real :: d_type + if (present(clock)) then ; if (clock>0) call cpu_clock_begin(clock) ; endif + call mpp_do_group_update(group, MOM_dom%mpp_domain, d_type, omp_offload) + if (present(clock)) then ; if (clock>0) call cpu_clock_end(clock) ; endif +end subroutine do_group_pass +``` + +`mpp_do_group_update` itself lives in the external FMS `mpp_domains` library (fetched at build time via +`ac/deps/Makefile.fms.in`, not vendored in this repo), so the actual GPU-aware implementation (CUDA- +aware MPI send/recv directly on device pointers, vs. staging through a host buffer) cannot be +inspected here — only the MOM-side call contract. + +> **Resolved (2026-07-14):** the FMS `omp_offload` path is a genuine device path with **no** fallback. +> In the sibling FMS checkout, `mpp_group_update.fh` device-packs halos (`target teams distribute … +> if(use_device_ptr)` into a device buffer) and `mpp_transmit_mpi.fh` posts `MPI_ISEND`/`IRECV` inside +> `!$omp target data use_device_ptr(...)` — real CUDA-aware MPI on device pointers. There is no +> capability check: a non-GPUDirect MPI stack means a crash or corruption, **not** graceful host +> staging. The nonblocking variants hardcode `use_device_ptr = .false. ! placeholder`, which confirms +> the gated/unconditional split in §4 from the FMS side. + +This parameter exists **only** in the FMS2 +infra shim; `config_src/infra/FMS1/MOM_domain_infra.F90:1144` still has the old +`do_group_pass(group, MOM_dom, clock)` signature with no `omp_offload`. `ac/configure.ac:238-241` +auto-selects FMS2 vs FMS1 based on whether the linked FMS provides `fms2_io_mod`, so a GPU build +implicitly requires a modern-enough FMS with the offload-aware `mpp_do_group_update` overload — this +is a build-time external dependency, not something visible in `src/`. + +`pass_var`/`pass_vector` (single-field, non-grouped passes) have **no** `omp_offload` parameter at +all (`MOM_domain_infra.F90:173,220`). Only the *batched, group* path was extended for GPU-awareness; +single-field passes remain host-mediated by construction (§4). + +### 2.2 What the change replaced — before/after in `MOM_barotropic.F90` + +The same commit (`656e09013`) simultaneously deleted a large amount of manual host-staging code in +`MOM_barotropic.F90`, which is the clearest illustration of what `omp_offload=.true.` buys. Before: + +```fortran +! before 656e09013 (illustrative, from the diff's "-" lines) +!$omp target update from(bt_rem_u, bt_rem_v, eta_src) +!$omp target update if(integral_BT_cont) from(eta_IC) +! ... 5 more conditional "from" staging lines ... +call do_group_pass(CS%pass_eta_bt_rem, CS%BT_Domain) +!$omp target update to(bt_rem_u, bt_rem_v, eta_src) +!$omp target update if(integral_BT_cont) to(eta_IC) +! ... 5 more conditional "to" staging lines ... +if (.not.use_BT_cont) then + !$omp target update from(Datu, Datv) + call do_group_pass(CS%pass_Dat_uv, CS%BT_Domain) + !$omp target update to(Datu, Datv) +endif +``` + +After: + +```fortran +! src/core/MOM_barotropic.F90:1772-1774 (dev/gpu) +call do_group_pass(CS%pass_eta_bt_rem, CS%BT_Domain, omp_offload=.true.) +if (.not.use_BT_cont) call do_group_pass(CS%pass_Dat_uv, CS%BT_Domain, omp_offload=.true.) +call do_group_pass(CS%pass_force_hbt0_Cor_ref, CS%BT_Domain, omp_offload=.true.) +``` + +Ten-plus `!$omp target update from/to` directives (each a D2H then H2D transfer of every field in the +group, done unconditionally around the pass) collapsed into three plain `omp_offload=.true.` calls. +The same pattern recurs at four more spots in the same commit/file (`:929/:1439/:1641/:1896` in the +pre-fix numbering). **Data-residency implication:** when `omp_offload=.true.`, the halo exchange reads +and writes the fields *in place on the device* — the arrays are expected to already be +`!$omp target enter data`-mapped, and the exchange does not require (and the MOM-side code no longer +performs) a round-trip through a host mirror. When the flag is absent/false (the nonblocking branches, +§4, and all bare `pass_var`/`pass_vector` calls), the exchange is host-mediated and the caller must +stage the data there-and-back manually with `!$omp target update from(...)` / `to(...)`. + +One TODO-turned-real-fix, in the same commit's diff at the inner sub-cycle loop +(`MOM_barotropic.F90:2757`, `btstep_timeloop`): + +```fortran +- ! TODO: direct GPU-to-GPU transfer +- !$omp target update from(ubt, vbt, eta) +- call do_group_pass(CS%pass_eta_ubt, CS%BT_Domain, clock=id_clock_pass_step) +- !$omp target update to(ubt, vbt, eta) ++ call do_group_pass(CS%pass_eta_ubt, CS%BT_Domain, clock=id_clock_pass_step, omp_offload=.true.) +``` + +— the barotropic inner-loop halo pass (the one amortized by the wide halo, §5) is exactly the "direct +GPU-to-GPU transfer" the TODO asked for. + +--- + +## 3. Every `omp_offload=.true.` call site + +`grep -rn omp_offload src/ config_src/` returns **29** lines: 3 belong to the single FMS2 +`do_group_pass` definition (the `subroutine` line `:1143`, the `omp_offload` argument declaration +`:1152`, and the `mpp_do_group_update` call `:1158` — all in `config_src/infra/FMS2/`, since FMS1 has +no such parameter), leaving **26** call sites (the architecture doc's "~25" estimate, §7.3). +*(Verified: `grep -rn omp_offload src/ config_src/ | wc -l` = 29.)* Grouped by module: + +| # | File:line | Group-pass handle | Domain | Fields (from `create_group_pass`) | Context / stencil | +|---|---|---|---|---|---| +| 1 | `MOM.F90:742` | `pass_tau_ustar_psurf` | `G%Domain` | `forces%taux,tauy` (`:732`), `ustar` (`:734`), `tau_mag` (`:736`), `p_surf` (`:738`) — the latter two only `if (associated(...))` | top of `step_MOM`; **gated** — this `omp_offload=.true.` blocking call is the `else` branch of `if (nonblocking_p_surf_update)` (`:739-743`). Note `:681` enter-data maps only `forces, forces%taux/tauy/ustar` (**not** `tau_mag`/`p_surf`) | +| 2 | `MOM.F90:2112` | `pass_uv_T_S_h` | `G%Domain` | `u,v` (`:2106`), `tv%T`,`tv%S` (`:2108,2110`), `h` (`:2111`) | `halo=dynamics_stencil = min(3,nihalo,njhalo)` (`:2105`); the "GPU-aware" pass cited in `00-architecture.md` §4.1 | +| 3 | `MOM_dynamics_split_RK2.F90:663` | `pass_eta` | `G%Domain` | `eta` (`halo=1`, `:506`) | predictor stage, blocking branch (paired with nonblocking start at `:580`/complete `:658`) | +| 4 | `MOM_dynamics_split_RK2.F90:664` | `pass_visc_rem` | `G%Domain` | `CS%visc_rem_u,visc_rem_v` (`:507`, `halo=max(1,cont_stencil)`) | predictor stage | +| 5 | `MOM_dynamics_split_RK2.F90:841` | `pass_visc_rem` | `G%Domain` | same as #4 | predictor stage, second occurrence (after `vertvisc_remnant`) | +| 6 | `MOM_dynamics_split_RK2.F90:847` | `pass_uvp` | `G%Domain` | `up,vp` (`:509`, `halo=max(1,cont_stencil)`) | blocking branch (nonblocking start `:829`, complete `:844`) | +| 7 | `MOM_dynamics_split_RK2.F90:860` | `pass_hp_uv` | `G%Domain` | `hp` (`halo=cor_stencil`), `u_av,v_av` and `uh,vh` (`halo=max(cor_stencil,vel_stencil)`) (`:512-514`) | after predictor `continuity` | +| 8 | `MOM_dynamics_split_RK2.F90:1131` | `pass_visc_rem` | `G%Domain` | same as #4 | corrector stage | +| 9 | `MOM_dynamics_split_RK2.F90:1137` | `pass_uv` | `G%Domain` | `u_inst,v_inst` (`:510`, `halo=max(2,cont_stencil)`) | blocking branch (nonblocking start `:1118`, complete `:1134`) | +| 10 | `MOM_dynamics_split_RK2.F90:1153` | `pass_h` | `G%Domain` | `h` (`:515`, `halo=max(cor_stencil,cont_stencil)`) | after corrector `continuity` | +| 11 | `MOM_dynamics_split_RK2.F90:1165` | `pass_av_uvh` | `G%domain` | `u_av,v_av`, `uh,vh` (`:516-517`, `halo=max(cor_stencil,vel_stencil)`) | blocking branch (nonblocking start `:1163`) | +| 12 | `MOM_barotropic.F90:1008` | `CS%pass_q_DCor` | `CS%BT_Domain` | `q` (position `CORNER`), `DCor_u,DCor_v` (`:822-823`) | wide-halo; blocking branch | +| 13 | `MOM_barotropic.F90:1550` | `CS%pass_gtot` | `CS%BT_Domain` | `gtot_E,gtot_N` / `gtot_W,gtot_S` (`:828-830`) | wide-halo | +| 14 | `MOM_barotropic.F90:1551` | `CS%pass_ubt_Cor` | `G%Domain` | `ubt_Cor,vbt_Cor` (`:862`) | normal halo | +| 15 | `MOM_barotropic.F90:1772` | `CS%pass_eta_bt_rem` | `CS%BT_Domain` | `eta_src`, `bt_rem_u/v`, `eta_PF*`, `eta_IC`, `dyn_coef_eta`, `Rayleigh_u/v` (`:834-848`) | wide-halo | +| 16 | `MOM_barotropic.F90:1773` | `CS%pass_Dat_uv` | `CS%BT_Domain` | `Datu,Datv` (`:857`) | wide-halo; only if `.not. use_BT_cont` | +| 17 | `MOM_barotropic.F90:1774` | `CS%pass_force_hbt0_Cor_ref` | `CS%BT_Domain` | `BT_force_u/v`, `uhbt0/vhbt0`, `Cor_ref_u/v` (`:853-855`) | wide-halo | +| 18 | `MOM_barotropic.F90:2022` | `CS%pass_e_anom` | `G%Domain` | `e_anom` (`:868`) | normal halo | +| 19 | `MOM_barotropic.F90:2070` | `CS%pass_ubta_uhbta` | `G%Domain` | `CS%ubtav,vbtav`, `uhbtav,vhbtav` (`:869-870`) | normal halo | +| 20 | `MOM_barotropic.F90:2757` | `CS%pass_eta_ubt` | `CS%BT_Domain` | `eta`, `ubt,vbt` (`:2740-2741`) | **the inner sub-cycle pass**, amortized by wide halos (§5) | +| 21 | `MOM_barotropic.F90:5290` | `BT_cont%pass_polarity_BT` | `BT_Domain` | `u_polarity,v_polarity`, `uBT_EE/vBT_NN`, `uBT_WW/vBT_SS` (`:5279-5281`) | face-area closure setup | +| 22 | `MOM_barotropic.F90:5291` | `BT_cont%pass_FA_uv` | `BT_Domain` | `FA_u_EE/FA_v_NN`, `FA_u_E0/FA_v_N0`, `FA_u_W0/FA_v_S0`, `FA_u_WW/FA_v_SS` (`:5283-5286`) | face-area closure setup | +| 23 | `MOM_tracer_advect.F90:277` | `CS%pass_uhr_vhr_t_hprev` | `G%Domain` | `uhr,vhr` (`:180`), `hprev` (`:181`), `Reg%Tr(m)%t` for each tracer (`:183`) | inner advection iteration `do itt=1,max_iter` | +| 24 | `MOM_tracer_hor_diff.F90:573` | `CS%pass_t` | `G%Domain` | `Reg%Tr(m)%t` per tracer (`:244`) | neutral-diffusion iteration | +| 25 | `MOM_tracer_hor_diff.F90:882` | `CS%pass_t` | `G%Domain` | same | mixed-layer/buffer-layer density-coordinate setup | +| 26 | `MOM_tracer_hor_diff.F90:1283` | `CS%pass_t` | `G%Domain` | same | along-surface diffusion iteration `do itt=1,num_itts` | + +**Corrected module counts** (recounted from the grep above — the fields sum to the 26 total): +**11** sites are in `MOM_barotropic.F90` (**8** on the wide-halo `BT_Domain` — rows 12,13,15,16,17,20 +plus rows 21,22 whose `BT_Domain` dummy is `CS%BT_Domain`, passed at `MOM_barotropic.F90:1217/1219`), +9 in `MOM_dynamics_split_RK2.F90`, 2 in `MOM.F90`, 1 in `MOM_tracer_advect.F90`, 3 in +`MOM_tracer_hor_diff.F90` ⇒ 11+9+2+1+3 = 26. *(An earlier draft said "15 (7 on `CS%BT_Domain`)"; that +was wrong — the table itself has only 11 barotropic rows, 8 of them wide-halo, and 15+9+2+1+3=30≠26.)* +See `09-barotropic-solver.md` §2.2 for a barotropic-only version of this table with additional +`btstep`-internal commentary. + +### 3.1 Where halos are still done on host + +Every call above is a **grouped** pass. Bare, single-field `pass_var`/`pass_vector` calls never carry +`omp_offload` (the parameter doesn't exist on that entry point, §2.1) and are therefore host-mediated +by default — if the field is device-resident, the caller must bracket the call with manual +`!$omp target update from(...)` / `to(...)`, or the pass simply operates on a host-only field never +mapped to device in the first place. Confirmed examples in modules the architecture doc lists as +"largely untouched" on `dev/gpu` (§6.3): + +``` +src/parameterizations/lateral/MOM_thickness_diffuse.F90:2234: call pass_var(CS%khth2d, G%domain) +src/parameterizations/lateral/MOM_mixed_layer_restrat.F90:365: call pass_var(mle_fl_2d, G%domain, halo=1) +src/parameterizations/lateral/MOM_mixed_layer_restrat.F90:680: call pass_var(h, G%domain, To_West+To_South+Omit_Corners, halo=1) +src/parameterizations/lateral/MOM_mixed_layer_restrat.F90:864: call pass_var(bflux, G%domain, halo=1) +src/parameterizations/lateral/MOM_mixed_layer_restrat.F90:1150: call pass_var(h, G%domain, To_West+To_South+Omit_Corners, halo=1) +src/parameterizations/lateral/MOM_mixed_layer_restrat.F90:1479: call pass_var(h, G%domain, To_West+To_South+Omit_Corners, halo=1) +src/parameterizations/lateral/MOM_mixed_layer_restrat.F90:1752: call pass_var(CS%MLD_Tfilt_space, G%domain) +src/parameterizations/lateral/MOM_mixed_layer_restrat.F90:1764: call pass_var(CS%Cr_space, G%domain) +src/parameterizations/lateral/MOM_mixed_layer_restrat.F90:1954-1956: pass_var(CS%MLD_filtered/_slow/wpup_filtered, G%domain) +``` + +`MOM_mixed_layer_restrat.F90` and `MOM_thickness_diffuse.F90` are both `+0`-diff (untouched) on +`dev/gpu` per `00-architecture.md` §6.3 — these `h`, `bflux`, `khth2d` halo updates run entirely on +the CPU today; if/when those modules are ported (branch `edoyango/port/thickness_diffuse`, +`bodner-naive-port`), these calls are exactly where `create_group_pass`/`omp_offload=.true.` would +need to be introduced to keep the halo exchange on-device. + +--- + +## 4. The nonblocking group-pass pattern (`start_group_pass`/`complete_group_pass`) + +`MOM_domains.F90:49` re-exports the non-blocking single-field entry points +(`pass_var_start/complete`, `pass_vector_start/complete`) and `:51` the non-blocking **group** entry +points used on the GPU-relevant hot path. Both are gated at runtime by `NONBLOCKING_UPDATES` +(`src/framework/MOM_domains.F90:215`, `G%nonblocking_updates = G%Domain%nonblocking_updates`, +`MOM_grid.F90:300`; in `MOM_barotropic.F90` the same flag is cached as `nonblock_setup = +G%nonblocking_updates`, `:789`). + +**Key finding: `start_group_pass`/`complete_group_pass` have no `omp_offload` parameter at all** +(`MOM_domain_infra.F90:1165,1186`) — only the blocking `do_group_pass` was extended. So whenever +`G%nonblocking_updates` is true, the code path is *not* GPU-aware and must stage data through the host +manually around the non-blocking send/receive, exactly like the pre-`656e09013` barotropic code +(§2.2). The pattern, repeated at every dycore group-pass site: + +```fortran +! src/core/MOM_dynamics_split_RK2.F90:656-665 +if (G%nonblocking_updates) then + call complete_group_pass(CS%pass_eta, G%Domain) + !$omp target update to(eta) + !$omp target update from(CS%visc_rem_u, CS%visc_rem_v) + call start_group_pass(CS%pass_visc_rem, G%Domain) +else + call do_group_pass(CS%pass_eta, G%Domain, omp_offload=.true.) + call do_group_pass(CS%pass_visc_rem, G%Domain, omp_offload=.true.) +endif +``` + +i.e. **the nonblocking path and the GPU-offload path are mutually exclusive branches of the same +`if`.** Where such an `if (G%nonblocking_updates)/(nonblock_setup)/(nonblocking_p_surf_update)` guard +exists, turning on `NONBLOCKING_UPDATES` reverts *that* site to host-staged communication (extra D2H +before `start_group_pass`, extra H2D after `complete_group_pass`), trading GPU-resident halo exchange +for compute/communication overlap on the host side. + +> **CORRECTED — scope of the revert.** An earlier draft claimed enabling `NONBLOCKING_UPDATES` +> "silently reverts **every one** of the 26 sites in §3." That is **not true**. Only the sites that +> physically sit in an `if (…nonblocking…)/else` block have a nonblocking counterpart; the rest are +> **unconditional** `do_group_pass(…, omp_offload=.true.)` calls that stay GPU-aware regardless of the +> flag. Recounted against the code: +> +> | | Gated (reverts to host-staging when `NONBLOCKING_UPDATES=.true.`) | Always `omp_offload=.true.` (flag-independent) | +> |---|---|---| +> | `MOM.F90` | `:742` `pass_tau_ustar_psurf` (else of `nonblocking_p_surf_update`, `:739`) | `:2112` `pass_uv_T_S_h` (no guard) | +> | `MOM_dynamics_split_RK2.F90` | `:663` `pass_eta`, `:664` `pass_visc_rem`, `:847` `pass_uvp`, `:1137` `pass_uv`, `:1165` `pass_av_uvh` | `:841` `pass_visc_rem`, `:860` `pass_hp_uv`, `:1131` `pass_visc_rem`, `:1153` `pass_h` | +> | `MOM_barotropic.F90` | `:1008` `pass_q_DCor` (else of `:1004`), `:1550` `pass_gtot`, `:1551` `pass_ubt_Cor` (else of `:1539`), `:1772/:1773/:1774` `pass_eta_bt_rem`/`pass_Dat_uv`/`pass_force_hbt0_Cor_ref` (else of `:1763`), `:2022` `pass_e_anom` (else of `:2017`), `:2070` `pass_ubta_uhbta` (else of `:2064`) | `:2757` `pass_eta_ubt` (inner sub-cycle, §5), `:5290` `pass_polarity_BT`, `:5291` `pass_FA_uv` | +> | `MOM_tracer_advect.F90` | — | `:277` `pass_uhr_vhr_t_hprev` (no nonblocking in file) | +> | `MOM_tracer_hor_diff.F90` | — | `:573`, `:882`, `:1283` `pass_t` (no nonblocking in file) | +> +> **14 gated, 12 unconditional (14+12 = 26).** So `NONBLOCKING_UPDATES` disables the GPU-aware path at +> the 14 gated sites only — notably the barotropic *inner sub-cycle* pass (`:2757`, the hottest +> exchange, §5) and **all four tracer passes** keep `omp_offload=.true.` no matter what. The flag +> therefore *weakens* GPU-residency for the outer dynamics/barotropic-setup exchanges but does **not** +> globally defeat GPU-aware exchange. + +One gated site (`MOM.F90:732-744`, `pass_tau_ustar_psurf`) is the exception that proves the staging +rule: its nonblocking branch (`:739-740`) has *no* explicit `target update from` before +`start_group_pass`, because `forces%taux/tauy/ustar` were only just +`!$omp target enter data map(to: ...)`'d at `:681` (that map covers `forces, forces%taux, forces%tauy, +forces%ustar` — **not** `tau_mag`/`p_surf`, which are host-resident here) — host and device copies are +still identical at that point, so starting the nonblocking send from the (still valid) host mirror +needs no extra staging. + +`MOM.F90:743`/`:818` (cited in the task) are exactly `pass_tau_ustar_psurf`'s +`do_group_pass(...,omp_offload=.true.)` (blocking branch, `:742`) and its nonblocking counterpart's +`complete_group_pass` (`:818`), gated by the same `nonblocking_p_surf_update` flag +(`:727-729`, a refinement of `G%nonblocking_updates` that also requires `p_surf`/`SpV_avg`/`T` not to +be simultaneously in play). + +**Conclusion for Q6:** the nonblocking API is preserved and actively used (it predates the GPU port +and is wired through *most* dynamics/barotropic group passes), but it is **not GPU-offload-aware** — it +is a CPU-communication-overlap feature that, when enabled, disables the `omp_offload` GPU-aware path +**at the 14 gated sites** (falling back to explicit host staging) while leaving the 12 unconditional +sites — including the barotropic inner sub-cycle and all tracer passes — on the GPU-resident path. + +--- + +## 5. The wide-halo barotropic domain — minimizing exchange frequency + +Full algorithmic treatment in `09-barotropic-solver.md` §1.2/§2.2; summary here for the +communication/multi-GPU angle. + +`CS%BT_Domain` (`MOM_barotropic.F90:338`) is created once, in `barotropic_init`, by cloning the normal +domain with a larger minimum halo: + +```fortran +! src/core/MOM_barotropic.F90:6103-6104 +! Initialize a version of the MOM domain that is specific to the barotropic solver. +call clone_MOM_domain(G%Domain, CS%BT_Domain, min_halo=wd_halos, symmetric=.true.) +``` + +`wd_halos` comes from the runtime parameters `BT_USE_WIDE_HALOS` (default `.true.`) and `BTHALO` +(minimum halo size, default 0 ⇒ under dynamic memory `wd_halos = bt_halo_sz` as configured; under +`STATIC_MEMORY_` it is fixed by the `WHALOI_`/`WHALOJ_` macros, `:47-51`, +`WHALOI_ = MAX(BTHALO_-NIHALO_, 0)`). `clone_MD_to_d2D` (`MOM_domain_infra.F90:1717-1780`) takes +`max(existing_halo, min_halo)` — the wide-halo domain has the **same PE layout / decomposition** as +`G%Domain`, only a bigger halo ring width. + +Why this matters for communication volume: `btstep_timeloop` (`MOM_barotropic.F90:2376`) runs +`nstep+nfilter` small barotropic sub-steps per call, each of which only needs a 1–2 point stencil +update of `eta`/`ubt`/`vbt`. Rather than issuing a halo exchange on every sub-step, the valid +(non-communicated) index range is allowed to **shrink by `stencil` points per sub-step** ("march +inward"), and a `do_group_pass(..., omp_offload=.true.)` is issued only once the valid range would no +longer cover the true computational domain: + +```fortran +! src/core/MOM_barotropic.F90:2621-2630, 2754-2758 +stencil = max(1, CS%min_stencil) +num_cycles = 1 +if (CS%use_wide_halos) & + num_cycles = min((is-CS%isdw) / stencil, (js-CS%jsdw) / stencil) +isvf = is - (num_cycles-1)*stencil ; ievf = ie + (num_cycles-1)*stencil +jsvf = js - (num_cycles-1)*stencil ; jevf = je + (num_cycles-1)*stencil +... +do n=1,nstep+nfilter + ... + if ((iev - stencil < ie) .or. (jev - stencil < je)) then + call do_group_pass(CS%pass_eta_ubt, CS%BT_Domain, clock=id_clock_pass_step, omp_offload=.true.) + isv = isvf ; iev = ievf ; jsv = jsvf ; jev = jevf + else + isv = isv - stencil ; iev = iev + stencil ; jsv = jsv - stencil ; jev = jev + stencil ! march inward, no comm + endif +enddo +``` + +With `num_cycles` sub-steps amortized per exchange, the number of `omp_offload=.true.` MPI/NVSHMEM- +style exchanges across the barotropic sub-cycle drops from `O(nstep)` to `O(nstep/num_cycles)`. This +is the single biggest lever for reducing communication (and hence device-buffer synchronization) +overhead in the whole dycore, because `nstep` (set by `set_dtbt`, `MOM_barotropic.F90:3797`, +cited in `00-architecture.md` §4.3) is typically O(10)–O(30) sub-cycles per outer dynamics step. Every +exchange on `CS%BT_Domain` in the table in §3 (rows 12,13,15,16,17,20,21,22) benefits from this same +wider halo even outside the inner sub-cycle loop — e.g. `pass_q_DCor` and `pass_eta_bt_rem` are +one-shot per `btstep` call but still use the wide halo so their downstream consumers (`btstep_timeloop` +itself) can march inward before needing another exchange. + +--- + +## 6. Device↔host transfer boundaries and data residency + +Putting §2–§4 together, the rule of thumb on `dev/gpu` is: + +- **Grouped, `omp_offload=.true.` passes** (§3, 26 sites): operate directly on device-resident arrays. + No corresponding `!$omp target update` is needed immediately around the call — the calling code is + expected to already have the fields `enter data`-mapped (typically once, at CS init or at the top of + `step_MOM`/`step_MOM_dyn_split_RK2`). +- **Nonblocking passes** (`start_group_pass`/`complete_group_pass`, §4): always host-mediated; + explicit `!$omp target update from(...)` precedes `start_group_pass` and `!$omp target update + to(...)` follows `complete_group_pass`, except where the host mirror is already known-valid + (the `pass_tau_ustar_psurf` case, §4). +- **Bare `pass_var`/`pass_vector`** (§3.1): host-mediated by construction; used almost exclusively in + modules that are still entirely CPU (`MOM_mixed_layer_restrat.F90`, `MOM_thickness_diffuse.F90`), + so in practice these fields (`h`, `bflux`, `khth2d`, MLD filters) simply never leave the host in the + first place inside those routines — no transfer is "forced" so much as the whole subroutine is + outside the mapped region. +- **`redistribute_array_*`/`global_field`/`broadcast_domain`** (`MOM_domain_infra.F90:1207-1261` and + neighboring routines, re-exported at `MOM_domains.F90:20,44`): used for domain-to-domain + redistribution (e.g. coarsening, I/O gather) — not touched by the GPU port at all, and by + construction operate on host arrays (`mpp_redistribute`, no offload argument), so any GPU-resident + field passed to them requires a manual `target update from` beforehand — none of the omp_offload + call sites in §3 route through this family. + +The forcing-field entry point at `step_MOM` is the cleanest illustration of the residency contract: +`forces%taux/tauy/ustar` arrive from the coupler on the host, are `!$omp target enter data map(to: +...)`'d exactly once (`MOM.F90:681`), and every subsequent group pass over them +(`pass_tau_ustar_psurf`, §3 row 1) runs `omp_offload=.true.` for the remainder of that `step_MOM` call +— the H2D transfer happens once per coupling step, not once per halo exchange. + +--- + +## 7. Multi-GPU: the `MOM_tracer_advect.F90` answer-change bugfix + +Branch `remotes/edoyango/bugfix-traceradvection-multigpu` (rebased commits `d28afbf32`, +`680f927e7`), merged into `dev/gpu` as `a774eb331` and `e182de310`. Both fix real correctness bugs +that **only manifest when the domain is split across more than one GPU** (i.e. more than one MPI rank, +each bound to its own device) — a single-GPU/single-rank run does not exercise the code paths that +expose either bug. `git show e182de310` / `git show a774eb331` for the full diffs; key excerpts below. + +### 7.1 Bug 1 — unreduced write to `domore_k(k)` (`e182de310`, "advect_tracer: fix multi gpu answer change") + +`advect_tracer` (`MOM_tracer_advect.F90`) tracks, per vertical layer `k`, whether any more advection +iterations are needed on this PE via an integer flag array `domore_k(1:nz)`. Before the fix: + +```fortran +domore_k(k) = 0 +do concurrent (j=jsv:jev, domore_u(j,k)) + domore_k(k) = 1 +enddo +do concurrent (J=jsv+stencil-1:jev-stencil, domore_v(J,k)) + domore_k(k) = 1 +enddo +``` + +Every iteration that fires writes the **same** value (`1`) to the **same** array element +`domore_k(k)`, with no `reduce`/`local` locality-spec on the `do concurrent`. Per ISO Fortran 2018 +semantics, a `do concurrent` construct must not have iterations that access the same variable unless +that access pattern is declared (`local`/`local_init`/`reduce`); an unguarded shared write like this +is technically nonconforming, and on `nvfortran`'s GPU lowering of `do concurrent` the compiler needs +an explicit `reduce` locality-spec to generate correct synchronized/atomic accumulation into a shared +scalar — without it, nothing guarantees the write from an arbitrary GPU thread actually lands in +global memory before the loop's implicit barrier, or that a value written by one thread block isn't +overwritten by another block's stale copy. The fix: + +```fortran +! e182de310, src/tracer/MOM_tracer_advect.F90 +domore_k_tmp = 0 +do concurrent (j=jsv:jev, domore_u(j,k)) DO_LOCALITY(reduce(max:domore_k_tmp)) + domore_k_tmp = 1 +enddo +do concurrent (J=jsv+stencil-1:jev-stencil, domore_v(J,k)) DO_LOCALITY(reduce(max:domore_k_tmp)) + domore_k_tmp = 1 +enddo +domore_k(k) = domore_k_tmp +``` + +using a **new scalar temporary** `domore_k_tmp` rather than the array element `domore_k(k)` directly, +because (per the commit message) "do concurrent can't use array elems yet" as a reduction target — +Fortran's `reduce` locality-spec (and `nvfortran`'s support for it, gated by +`HAVE_FC_DO_CONCURRENT_LOCAL` / the `DO_LOCALITY` macro, `src/framework/do_concurrent_compat.h`) only +accepts scalar variables. Three call sites in `advect_tracer` had this pattern +(`:298-311`, `:335-347`, `:362-374` in the pre-fix file); all three were converted identically. The +same commit also removed now-redundant staging (`!$omp target` bracket around a plain +`domore_k(k) = 0` reset, and a `!$omp target update from(domore_k)` before the `sum_across_PEs` +reduction across PEs) since `domore_k` no longer needs a manual round trip once the device-side +reduction is correct. + +**Why only multi-GPU:** the write pattern is undefined-behavior-adjacent on *any* GPU execution, but +whether it produces a wrong answer depends on how many independent thread blocks/teams the compiler +launches to cover the `j`/`J` range and whether their partial results are ever lost before the implicit +end-of-loop synchronization. A single-GPU (single-rank) run's per-rank `jsv:jev` range spans the whole +(large) global domain, likely scheduled by the runtime as one arrangement of teams the compiler +happens to handle correctly (or the CPU fallback masks it entirely). A multi-GPU run decomposes the +domain into many smaller per-rank tiles — different loop trip counts, different team/block counts per +kernel launch — which is exactly the situation where a missing `reduce` clause is more likely to +surface as a dropped update (`domore_k(k)` silently staying `0` when it should be `1`), causing +`advect_tracer` to terminate its outward iteration early on some layers/PEs and silently +under-advecting a tracer — an "answer change" that differs by GPU count rather than being wrong on +every run. (This mechanism is inferred from the code and Fortran locality semantics; the commit +message states only the symptom — "leading to answer changes on multiple GPUs" — and the fix, not the +precise compiler-internal cause.) + +### 7.2 Bug 2 — `Reg%Tr(:)` mapped `alloc` instead of `to` (`a774eb331`, "tracer_advect: fix map of Reg%Tr(:)") + +```fortran +! before (e182de310's state) +!$omp target enter data map(to: OBC) map(alloc: domore_u, domore_v, uhr, vhr, uh_neglect, & +!$omp vh_neglect, hprev, local_advect_scheme, Reg, Reg%Tr(:)) +! after (a774eb331) +!$omp target enter data map(to: OBC, Reg, Reg%Tr(:)) map(alloc: domore_u, domore_v, uhr, vhr, uh_neglect, & +!$omp vh_neglect, hprev, local_advect_scheme) +``` + +`Reg` (the tracer registry, `tracer_registry_type`) and its array-of-derived-type component `Reg%Tr(:)` +(one `tracer_type` per registered tracer) were being `map(alloc:)`'d — i.e. the device gets freshly +allocated, **uninitialized** device memory for the registry structure, with no host→device copy. But +`advect_tracer` reads host-set scalar members of `Reg%Tr(m)` **inside** a `do concurrent` immediately +after this enter-data region: + +```fortran +! MOM_tracer_advect.F90:149-152 +do concurrent (m = 1:ntr) + local_advect_scheme(m) = Reg%Tr(m)%advect_scheme ! <-- reads a host-set scalar member + if (local_advect_scheme(m) < 0) local_advect_scheme(m) = CS%default_advect_scheme + ... +``` + +`Reg%Tr(m)%advect_scheme` is set once at tracer registration time on the host and never written on +device — with `map(alloc:)`, the device's copy of this scalar is whatever garbage happened to occupy +that freshly-allocated device memory, not the registered value. With `map(to:)`, it is the correct +host value copied down. (The commit's companion `map(release:)` vs `map(from:)` cleanup on the exit +side, and moving `hprev` from a `from`-mapped release to a plain `release`, are related tidying of the +same enter/exit-data region but not the correctness fix itself.) + +**Why only multi-GPU:** whether stale/garbage device memory for `advect_scheme` "happens to" produce +the right branch outcome depends on whatever was previously resident in that memory region on that +specific device — a function of allocator history, prior kernel launches, and each GPU's own +allocation pool. A single-GPU/single-rank test run exercises exactly one allocator instance, and it is +plausible for it to coincidentally return zeroed or otherwise-benign memory (e.g. a freshly-allocated +region on a lightly used device, or a value that happens to be a valid enum member of +`ADVECT_PLM/PPM/PPMH3`). Once the same code runs across multiple ranks/GPUs, each device's allocator +has an independent (and generally different) history, so the garbage value read back differs **per +GPU** — producing inter-rank inconsistency in `local_advect_scheme`, hence a different advection +scheme selected on different PEs for what should be the same tracer, hence a genuine multi-GPU-only +answer change (and, being a garbage-memory read, potentially also nondeterministic run-to-run). +Notably this fix is co-authored by the current user (`Co-authored-by: Jorge Luis Gálvez Vallejo +` in the `a774eb331` commit trailer). + +--- + +## 8. Summary answers + +1. **Domain/halo model:** computational (`isc:iec`) vs. data (`isd:ied` = computational + `NIHALO_=2` + halo) domains, symmetric-memory `IsdB=isd-1` B-grid offset, `create_group_pass` batches many fields + onto one `group_pass_type` handle so `do_group_pass` issues a single exchange per group; halo width + per pass is tailored to the consuming stencil (`cor_stencil`/`vel_stencil`/`cont_stencil`), not + always the full `NIHALO_`. +2. **`omp_offload` end-to-end:** `do_group_pass`'s optional `omp_offload` (`MOM_domain_infra.F90:1143`) + forwards straight to FMS's `mpp_do_group_update(...,omp_offload)` (external library, not in this + repo); when true, the exchange reads/writes device-resident buffers directly (no host round-trip); + commit `656e09013` shows the conversion from ~10 manual `!$omp target update from/to` pairs around + a bare `do_group_pass` to a single `omp_offload=.true.` call at 5+ sites in `MOM_barotropic.F90`, + including replacing a literal `! TODO: direct GPU-to-GPU transfer` comment. +3. **26 `omp_offload=.true.` call sites** across `MOM.F90` (2), `MOM_dynamics_split_RK2.F90` (9), + `MOM_barotropic.F90` (**11**, **8** of them on the wide-halo `BT_Domain`), `MOM_tracer_advect.F90` + (1), `MOM_tracer_hor_diff.F90` (3) — full table in §3. Bare `pass_var`/`pass_vector` (no + `omp_offload` parameter exists on that entry point) remain host-only, concentrated in the + still-unported `MOM_mixed_layer_restrat.F90` and `MOM_thickness_diffuse.F90`. +4. **Multi-GPU bug:** two independent, sequential fixes in `MOM_tracer_advect.F90` — (a) `domore_k(k)` + written from multiple `do concurrent` iterations without a `reduce` locality-spec, fixed by + reducing into a scalar temporary (`reduce(max:domore_k_tmp)`) because array elements aren't valid + `do concurrent` reduction targets yet; (b) `Reg`/`Reg%Tr(:)` mapped `alloc` instead of `to`, so a + host-set scalar (`Reg%Tr(m)%advect_scheme`) read inside a device `do concurrent` got uninitialized + device memory instead of its registered value. Both are latent on any GPU run but only produce + observable answer changes when multiple independent devices/allocators are involved (more/different + team-launch configurations for bug (a); independent per-device garbage-memory contents for bug (b)). +5. **Device↔host boundaries:** forced at (i) coupler ingest (`forces%*` `enter data` once per + `step_MOM`, `MOM.F90:681`), (ii) every nonblocking group pass (`start_group_pass`/ + `complete_group_pass` have no `omp_offload` — always host-staged), (iii) every bare + `pass_var`/`pass_vector` call, and (iv) redistribution/global-field routines (host-only, untouched + by the port). The wide-halo `CS%BT_Domain` (cloned via `clone_MOM_domain(..., min_halo=wd_halos)`, + `MOM_barotropic.F90:6104`) minimizes *how often* GPU-aware exchanges happen in the barotropic + sub-cycle by letting `btstep_timeloop` "march inward" for `num_cycles` sub-steps between + `do_group_pass(..., omp_offload=.true.)` calls, turning `O(nstep)` exchanges into + `O(nstep/num_cycles)`. +6. **Nonblocking pattern:** `start_group_pass`/`complete_group_pass` are preserved and wired to *many* + dycore/barotropic group passes, gated by runtime flag `NONBLOCKING_UPDATES` / + `G%nonblocking_updates` (`nonblock_setup` in barotropic), but they have no offload awareness — + enabling them reverts the **14 gated** sites (of 26) to manual host staging (`!$omp target update + from` before `start`, `to` after `complete`), the opposite of the GPU-resident `omp_offload=.true.` + blocking path in the same `if/else`. The other **12 sites** — including the barotropic inner + sub-cycle (`:2757`) and all four tracer passes — are unconditional `do_group_pass(…, + omp_offload=.true.)` calls unaffected by the flag (breakdown table in §4). + +--- + +## 9. Prescriptive rules for a porting agent + +Distilled from §§1–8 and cross-checked against the merged commits. Every rule is grounded in a +citation you can re-open. + +### 9.1 You are porting a module that calls `pass_var`/`pass_vector` + +These single-field entry points have **no** `omp_offload` parameter (`MOM_domain_infra.F90:173` +`pass_var_3d`, `:220` `pass_var_2d`, `:516/:662` `pass_vector_*` — none take it, and FMS1 lacks the +group offload arg entirely, `FMS1/MOM_domain_infra.F90:1144`). If the field the pass touches becomes +device-resident, you have two options: + +- **Option A — leave it host-staged, bracket the pass.** Keep the `call pass_var(field, G%domain, + …)` and wrap it: `!$omp target update from(field)` immediately before, `!$omp target update + to(field)` immediately after. This is exactly what the pre-`656e09013` barotropic code did (§2.2) + and what every *nonblocking* branch still does (§4). *Precondition:* `field` is already + `enter data`-mapped. *When to prefer:* a one-off pass, a rarely-hit path, or a field only + transiently on device — the round-trip cost is paid once and the diff stays tiny. + +- **Option B — promote to a grouped, offload-aware pass.** Add a `type(group_pass_type)` handle as a + CS member (e.g. `CS%pass_x`), register the field(s) with `create_group_pass(CS%pass_x, field, + G%Domain, halo=)` (batch several fields onto the *same* handle to get one MPI message, + §1.2), then replace the pass with `call do_group_pass(CS%pass_x, G%Domain, omp_offload=.true.)`. + The fields are then exchanged in place on the device with no host round-trip (§2.2, §6). + *Preconditions:* (i) the fields are `!$omp target enter data`-mapped for the lifetime of the pass; + (ii) the build links an FMS whose `mpp_do_group_update` accepts the offload flag — this is implicit + in any FMS2 build (`ac/configure.ac:238-241` selects FMS2 when `fms2_io_mod` is present) but is an + **external** dependency not vendored here (§2.1); (iii) if you place the offload call in an + `if (G%nonblocking_updates)/else`, remember the nonblocking branch is **not** offload-aware and must + still be hand-staged (§4) — or omit the guard entirely (as the tracer passes and barotropic inner + sub-cycle do) to stay unconditionally GPU-aware. *When to prefer:* a hot-path pass on fields that + live on-device across the whole routine — this is the blessed pattern (all 26 sites in §3). + +### 9.2 Halo-width-vs-stencil rule + +Pass `halo=` sized to the *consuming* kernel's stencil, never reflexively the full `NIHALO_=2`. +The dynamics driver computes `cor_stencil`/`vel_stencil`/`cont_stencil` once +(`MOM_dynamics_split_RK2.F90:498-504`) and each `create_group_pass` requests `halo=max(...)` of only +the stencils that read *that* field (`:506-517`); `pass_eta` uses `halo=1` because only a 1-point +stencil reads `eta`'s halo. Smaller halo ⇒ smaller message ⇒ less data crossing the PE/PE (and, when +`omp_offload`, device) boundary (§1.2). Corollary: if your ported kernel widens a stencil, widen the +matching `create_group_pass` `halo=` or you will read stale halo cells. + +### 9.3 `map(to:)` (not `map(alloc:)`) for registry-like structs read on device + +Any derived type — especially an **array-of-derived-type** component — whose *host-set scalar members* +are read inside a device region must be `map(to:)`, so the host values are copied down; +`map(alloc:)` gives the device freshly-allocated **uninitialized** memory. This is precisely the +`a774eb331` bug: `Reg`/`Reg%Tr(:)` were `map(alloc:)`'d, and `Reg%Tr(m)%advect_scheme` (set once at +registration, never on device) was read in a `do concurrent` at `MOM_tracer_advect.F90:149-152`, +yielding garbage (§7.2). The fix moved `Reg, Reg%Tr(:)` into the `map(to:)` clause. Rule: *if a device +loop reads it and the host wrote it, it is `to`, not `alloc`.* + +### 9.4 `reduce` locality-spec for shared-scalar accumulation in `do concurrent` + +A `do concurrent` whose iterations all write the same scalar/array-element needs an explicit +`DO_LOCALITY(reduce(op:var))` locality-spec, or `nvfortran`'s GPU lowering may drop updates (the +`e182de310` `domore_k` multi-GPU answer-change, §7.1). Two sub-rules from that fix: (i) the reduction +target must be a **scalar** — array elements like `domore_k(k)` are not yet valid `reduce` targets, so +reduce into a scalar temp and assign back (`domore_k(k) = domore_k_tmp`); (ii) once the device-side +reduction is correct, drop the now-redundant `!$omp target update from(...)` staging that previously +existed only to let the host recompute the value (`e182de310` also removed `domore_k` from the +`enter data` map). + +### 9.5 Why these two classes of bug only surface multi-GPU + +Both `MOM_tracer_advect.F90` fixes are latent on *any* GPU run but only change answers across +multiple devices (§7). The porting lesson: **single-GPU correctness does not prove a port correct.** +A missing `reduce` (9.4) depends on the team/block launch geometry, which changes with per-rank tile +size; an `alloc`-vs-`to` slip (9.3) depends on per-device allocator history. Validate ports at +≥2 ranks/GPUs, and compare `MOM_checksums` (`00-architecture.md` §7.2) across GPU counts, not just +CPU-vs-single-GPU. + +--- + +## Verification notes + +Verified against `dev/gpu` source and git (baseline `dev-gfdl`) on 2026-07-14. No code was built or +run. Line/commit references below were re-derived independently of the original draft. + +### Confirmed (spot-checked in code/git, correct as written) + +- **Domain/halo model.** `IsdB = isd-1` in symmetric mode: `MOM_hor_index.F90:90-96` (assignment + `HI%IsdB = HI%isd` then `if (HI%symmetric) HI%IsdB = HI%isd-1` at `:92-95`) — confirmed. `NIHALO_=2` + default (per `00-architecture.md` §3.1) — consistent. +- **Group-pass batching.** `create_group_pass` generic + `mpp_create_group_update`/ + `mpp_reset_group_update_field` append-on-reinit mechanics — confirmed at + `MOM_domain_infra.F90` (create routines) and the `MOM_dynamics_split_RK2.F90:498-517` stencil-sized + registration (halos `cor_/vel_/cont_stencil`, `pass_eta` `halo=1`) — all confirmed verbatim. +- **`omp_offload` plumbing.** FMS2 `do_group_pass(group, MOM_dom, clock, omp_offload)` at `:1143` + forwards to `mpp_do_group_update(..., omp_offload)` at `:1158` — confirmed. `start_group_pass` + (`:1165`) / `complete_group_pass` (`:1186`) and `pass_var_3d`/`pass_var_2d` (`:173`/`:220`), + `pass_vector_*` (`:516`/`:662`) take **no** `omp_offload` — confirmed. FMS1 `do_group_pass` (`:1144`) + has the old 3-arg signature — confirmed. +- **Commit `656e09013`.** Subject, and that it is the *only* substantive commit touching + `FMS2/MOM_domain_infra.F90` between `dev-gfdl` and `dev/gpu` (besides a relicense and a `dev/gfdl` + merge) — confirmed via `git log`. Its barotropic before/after (staging→`omp_offload`) at pre-fix + hunks `:929/:1439/:1641/:1896` (plus `:2578` btstep_timeloop TODO and `:5023` + set_local_BT_cont_types) — confirmed in the diff. +- **Multi-GPU bugfixes.** `e182de310` (`domore_k` → scalar `domore_k_tmp` with + `DO_LOCALITY(reduce(max:...))`, `domore_k` dropped from the `enter data` map) and `a774eb331` + (`Reg, Reg%Tr(:)` moved `map(alloc:)`→`map(to:)`, `hprev` `from`→`release`) — both diffs confirmed + verbatim; `e182de310` is an ancestor of `a774eb331` (order in §7 correct); Jorge Gálvez co-author + trailer on `a774eb331` confirmed. +- **Wide-halo clone + march-inward.** `clone_MOM_domain(G%Domain, CS%BT_Domain, min_halo=wd_halos, + symmetric=.true.)` at `MOM_barotropic.F90:6103-6104`, and the `btstep_timeloop` march-inward / + `do_group_pass(CS%pass_eta_ubt,...,omp_offload=.true.)` at `:2757` — confirmed. +- **`ac/configure.ac` FMS selection.** `AX_FC_CHECK_MODULE([fms2_io_mod], ...)` selecting FMS2 vs FMS1 + at `:238-241` — confirmed (draft's citation accurate). +- **Critical-claim mechanism.** `start/complete_group_pass` lack `omp_offload` and sit in the opposite + branch of `if (G%nonblocking_updates)` from the `omp_offload=.true.` `do_group_pass` — **confirmed** + verbatim at `MOM_dynamics_split_RK2.F90:657-665` and the barotropic `if (nonblock_setup)/else` blocks. + +### Corrected + +1. **Barotropic site count.** Draft said "**15** sites in `MOM_barotropic.F90` (**7** on + `CS%BT_Domain`)". `grep` shows **11** barotropic call sites, **8** of them wide-halo (`BT_Domain`); + 11+9+2+1+3 = 26 (the draft's 15+9+2+1+3 = 30 ≠ 26 was internally inconsistent, and its own §5 lists + 8 wide-halo rows). Fixed in §3 summary and §8.3. (Rows 21/22's `BT_Domain` dummy = `CS%BT_Domain`, + passed at `:1217/1219`, so they *are* wide-halo.) +2. **Scope of the `NONBLOCKING_UPDATES` revert (the strongest correction).** Draft said enabling it + "silently reverts **every one** of the 26 sites." Recounting against the code: only **14** sites + sit in an `if(…nonblocking…)/else` guard and revert to host staging; the other **12** are + **unconditional** `do_group_pass(…, omp_offload=.true.)` calls (dynamics `:841/:860/:1131/:1153`, + `MOM.F90:2112`, barotropic inner-sub-cycle `:2757` and setup `:5290/:5291`, and **all four** tracer + passes) that stay GPU-aware regardless of the flag. Rewrote §4 (with a gated-vs-unconditional + table), and the §4 conclusion / §8.6. +3. **`grep` accounting.** Draft: "the two infra definitions plus 26 call sites." It is **one** FMS2 + `do_group_pass` definition spanning **3** grep-matched lines (`:1143/:1152/:1158`) + 26 call sites = + 29 lines. Clarified in §3. +4. **`:681` enter-data contents.** Draft implied `forces%…/p_surf` are all mapped at `MOM.F90:681`. + That line maps only `forces, forces%taux, forces%tauy, forces%ustar` — **not** `tau_mag`/`p_surf`. + Corrected in §3 row 1 and §4. + +### Enhancements added + +- New **§9 "Prescriptive rules for a porting agent"**: (9.1) two options for porting a + `pass_var`/`pass_vector` module — host-staged bracket vs. promote-to-grouped-offload — with + preconditions each; (9.2) halo-width-vs-stencil rule; (9.3) `map(to:)`-not-`map(alloc:)` rule for + registry-like structs; (9.4) `reduce` locality-spec rule (scalar-temp workaround); (9.5) why single- + GPU correctness is insufficient. All grounded in the citations verified above. + +### Confidence + +**High** for everything checked directly against source/git (all §§1–7 code excerpts, the two commits, +the 26-site table, the gated/unconditional split, the corrected counts). Also **high** for the claims +resting on the external FMS library: the `omp_offload` device path has since been verified against the +FMS source (§2.1). **Medium** for the *inferred* "why only multi-GPU" +causal mechanisms in §7, which the draft already flags as inference from Fortran/compiler semantics +rather than from commit messages — that framing is appropriate and left as-is. diff --git a/knowledge/gpu-knowledge/12-diagnostics-io.md b/knowledge/gpu-knowledge/12-diagnostics-io.md new file mode 100644 index 0000000..c017031 --- /dev/null +++ b/knowledge/gpu-knowledge/12-diagnostics-io.md @@ -0,0 +1,573 @@ +# Diagnostics / IO Path and the Remaining Host-Only Surface (dev/gpu) + +> **Purpose.** The diagnostics/IO stack (`MOM_diag_mediator`, `MOM_diag_remap`, restarts) is the +> largest *unported, host-only* subsystem still touched every timestep. Every posted diagnostic and +> every restart write forces the model state across the PCIe/NVLink boundary. This document maps +> the `post_data` call path, catalogues the guarded `!$omp target update from(...)` transfer sites +> that keep those crossings conditional, evaluates the in-flight `diag_map_mediator_port` branch, +> and covers profiling (nvtx-on-clocks) and restart IO. Read `00-architecture.md` §4.1 and §7.4 +> first. + +--- + +## 1. The diagnostics path — confirmed host-only mediator + +`src/framework/MOM_diag_mediator.F90` is **byte-for-byte unchanged on `dev/gpu`**: + +``` +$ git diff --stat dev-gfdl...dev/gpu -- src/framework/MOM_diag_mediator.F90 +(empty) +``` + +`src/framework/MOM_restart.F90` is likewise **completely unchanged** (empty diff, zero directives) +— the restart registry is fully host-staged; see §6. + +### 1.1 Call map + +- **`post_data` generic** (`MOM_diag_mediator.F90:73-75`) dispatches on rank: + `module procedure post_data_3d, post_data_2d, post_data_1d_k, post_data_0d`. +- **`post_data_2d`** (`:1408`) → asserts a valid registered id, then loops the diag "variants" + linked list (CMOR aliases etc.) calling **`post_data_2d_low`** (`:1436`) for each. +- **`post_data_3d`** (`:1585`) → same pattern → **`post_data_3d_low`** (`:1750`). +- `post_data_2d_low`/`post_data_3d_low` do unit conversion (`field * diag%conversion_factor` into a + freshly host-`allocate`d `locfield`), optional masking, optional downsampling + (`downsample_diag_field`), vertical remapping via `diag_remap_do_remap` (in + `MOM_diag_remap.F90`), and finally `send_data_infra` (→ FMS `diag_manager`, host-only by + construction — FMS has no device awareness in this fork). +- `post_data_3d_by_column` (`:1927`) / `post_data_3d_by_point` (`:1945`) / `post_data_3d_final` + (`:1964`) are narrower host-side entry points used by column physics for point/column diagnostics. +- **`diag_update_remap_grids`** (`:3655-3759`) — snapshots `h`/`T`/`S` (or `alt_h/alt_T/alt_S`) as + plain Fortran pointer aliases (`h_diag => diag_cs%h`) and drives per-coordinate remap-grid + updates; **no OpenMP anywhere in the routine** — pure host pointer arithmetic and (per-coordinate) + calls into `MOM_diag_remap`. +- **`diag_copy_diag_to_storage`** (`:4130-4146`) / **`diag_copy_storage_to_diag`** (`:4149-4164`) — + plain whole-array Fortran assignment (`grid_storage%h_state(:,:,:) = h_state(:,:,:)`), host-side, + used to snapshot/restore the diagnostic grid across the dynamics/thermo sync boundary + (`MOM.F90:1100` calls `diag_copy_diag_to_storage(CS%diag_pre_sync, h, CS%diag)`). +- **`calculate_diagnostic_fields`** (`MOM_diagnostics.F90`) is the big host-side diagnostics + computation entered from `MOM.F90:1096`; it fans out to dozens of `post_data`/`post_product_u`/ + `post_product_v` calls guarded individually by `if (CS%id_xxx > 0)` (see e.g. lines 305-320 for + `id_u`, `id_v`, `id_h`, `id_usq`, `id_vsq`, `id_uv`). +- **`post_transport_diagnostics`** (`MOM_diagnostics.F90:1822`) posts transport diagnostics + (`umo`, `vmo`, dynamics `h`-tendency) after remapping storage is restored. + +### 1.2 The implication (Q1 answer) + +Because `MOM_diag_mediator.F90` has **zero** OpenMP directives, every array handed to `post_data_*` +is assumed **already host-resident**. Since the prognostic state (`u,v,h,uh,vh,uhtr,vhtr,T,S,...`) +lives device-resident for essentially the whole timestep (§4.2 of `00-architecture.md`), *the +transfer burden is pushed onto the caller*: each producing module must do its own +`!$omp target update from(...)` immediately before invoking `post_data`, or (as `MOM.F90:1091` does) +the driver does one blanket transfer of the whole synchronized state right before +`calculate_diagnostic_fields` is entered. **Posting any diagnostic is a device→host transfer either +way** — the only design freedom is *how much* is transferred and *how often* (whole-state blanket +vs. per-field guarded). + +--- + +## 2. Transfer-audit: guarded vs. unconditional `!$omp target update from(...)` + +`grep -rn "omp target update from" src/` returns **249** sites across `src/`. The large majority are +**not** diagnostics-specific — they cross OpenMP-target-region boundaries inside the dycore/barotropic +solver for algorithmic reasons (e.g. handing a partial result to a subsequent host-computed group +halo pass) or are unconditional `CS%debug`-class checksum staging. Only a **minority are explicitly +gated on a diagnostic id** (`if (CS%id_... > 0)`), which is the pattern this document is asked to +audit. Per-file raw counts of `target update from`: `MOM.F90` 56, `MOM_barotropic.F90` 52, +`MOM_hor_visc.F90` 53, `MOM_dynamics_split_RK2.F90` 45, `MOM_CoriolisAdv.F90` 11, +`MOM_set_viscosity.F90` 7, `MOM_tracer_hor_diff.F90` 6, `MOM_PressureForce_FV.F90` 5, +`MOM_vert_friction.F90` 4, `MOM_interface_heights.F90` 3, `MOM_diagnostics.F90` 3, +`MOM_lateral_mixing_coeffs.F90` 2, `MOM_tracer_advect.F90` 1, `MOM_state_initialization.F90` 1. + +### 2.1 Diagnostic-ID-guarded sites (the "only transfer if a diagnostic is active" pattern) + +| Site | Fields | Guard condition | +|---|---|---| +| `MOM_tracer_hor_diff.F90:722` | `khdt_x, khdt_y` | `if(CS%debug .or. CS%id_khdt_x>0 .or. CS%id_khdt_y>0)` | +| `MOM_diagnostics.F90:1825` | `uhtr` | `if (any([IDs%id_umo_2d, IDs%id_umo, IDs%id_uhtr] > 0))` | +| `MOM_diagnostics.F90:1826` | `vhtr` | `if (any([IDs%id_vmo_2d, IDs%id_vmo, IDs%id_vhtr] > 0))` | +| `MOM_diagnostics.F90:1827` | `h` | `if (IDs%id_dynamics_h_tendency > 0)` | +| `MOM_PressureForce_FV.F90:1301` | `e` (interface heights) | `if ((use_ALE .and. CS%Recon_Scheme > 0) .or. ...)` (debug/diag combo) | + +Quoted excerpt (`MOM_tracer_hor_diff.F90:719-733`): +```fortran +call post_data(CS%id_KhTr_h, Kh_h, CS%diag) +endif + +!$omp target update from(khdt_x, khdt_y) if(CS%debug .or. CS%id_khdt_x>0 .or. CS%id_khdt_y>0) +!$omp target exit data map(release: khdt_x, khdt_y, Kh_u, Kh_v) map(release: CS) + +if (CS%debug) then + call uvchksum("After tracer diffusion khdt_[xy]", khdt_x, khdt_y, ...) +endif + +if (CS%id_khdt_x > 0) call post_data(CS%id_khdt_x, khdt_x, CS%diag) +if (CS%id_khdt_y > 0) call post_data(CS%id_khdt_y, khdt_y, CS%diag) +``` +The transfer and the `post_data` call are **decoupled**: the `target update from` fires once +(covering both the checksum debug path and either diag id), then the two `if (CS%id_...>0)` guards +individually decide whether to actually call `post_data`. This avoids doing the transfer twice. + +Quoted excerpt (`MOM_diagnostics.F90:1822-1831`, `post_transport_diagnostics`): +```fortran +call diag_save_grids(diag) +call diag_copy_storage_to_diag(diag, diag_pre_dyn) + +!$omp target update from(uhtr) if (any([IDs%id_umo_2d, IDs%id_umo, IDs%id_uhtr] > 0)) +!$omp target update from(vhtr) if (any([IDs%id_vmo_2d, IDs%id_vmo, IDs%id_vhtr] > 0)) +!$omp target update from(h) if (IDs%id_dynamics_h_tendency > 0) + +if (IDs%id_umo_2d > 0) then + umo2d(:,:) = 0.0 + do k=1,nz ; do j=js,je ; do I=is-1,ie +``` +Here a **single** `uhtr`/`vhtr`/`h` transfer covers *several* downstream diagnostics (`umo_2d`, +`umo`, `uhtr` raw output) that would otherwise each want their own guard — the `any([...]>0)` +collapses multiple ids into one gate. + +`MOM_diagnostics.F90:967-975` (`calculate_vertical_integrals`) shows the same idea applied to a +derived (not raw-state) field: +```fortran +if (CS%id_col_ht > 0) then + !$omp target update to(h) + !$omp target enter data map(alloc: z_top) + call find_eta(h, tv, G, GV, US, z_top) + !$omp target exit data map(from: z_top) +``` +Note the direction here is `to(h)` (host→device, pushing possibly-stale-on-device `h` up) followed +by a device-computed `find_eta` and a `from: z_top` pull — the transfer is bidirectional around a +single-diagnostic-only device kernel. + +### 2.2 Unconditional / blanket transfer sites (the dominant pattern in practice) + +These are **not** individually diagnostic-gated; they transfer the whole synchronized prognostic +state once per relevant call so that every downstream `if (id>0)` check inside +`calculate_diagnostic_fields`/`post_*` sees host-valid data, trading a larger transfer for much +simpler code: + +- `MOM.F90:1091` — before `calculate_diagnostic_fields`: + ```fortran + if (MOM_state_is_synchronized(CS)) then + !$omp target update from(u, v, h, CS%uhtr, CS%vhtr) + call cpu_clock_begin(id_clock_other) ; call cpu_clock_begin(id_clock_diagnostics) + call enable_averages(CS%t_dyn_rel_diag, Time_local, CS%diag) + call calculate_diagnostic_fields(u, v, h, CS%uh, CS%vh, CS%tv, CS%ADp, & + CS%CDp, p_surf, CS%t_dyn_rel_diag, CS%diag_pre_sync, & + G, GV, US, CS%diagnostics_CSp) + ``` + This single line covers the entire fan-out of dozens of `if (CS%id_xxx>0) call post_data(...)` + branches inside `calculate_diagnostic_fields` — cheaper to reason about than gating each one, at + the cost of always paying for the transfer whenever the state is synchronized (every coupling + step boundary), whether or not *any* diagnostic in that big list is actually active. +- `MOM.F90:1113` — a **dead/disabled** duplicate of the same transfer, commented out with `!**`: + `!**!$omp target update from(u, v, h, CS%uhtr, CS%vhtr)` with a `TODO: This appears safe to remove + but needs verification.` — evidence the team is actively trying to prune redundant blanket + transfers. +- `MOM.F90:1036/1038` — around ALE remap/regrid (host-only stack, §4): `target update from(u,v,h)` + before `ALE_regridding_and_remapping`, `target update to(u,v,h)` after — this is **not** + diagnostics-related but is the same idiom (bracket a host-only region with from/to). +- `MOM.F90:1043-1046` (commit `ff86497d5`, already merged into `dev/gpu`) — see §2.3. +- `MOM_barotropic.F90` / `MOM_dynamics_split_RK2.F90` / `MOM_hor_visc.F90` / `MOM_CoriolisAdv.F90`: + their ~50 `target update from` sites each are almost entirely **algorithmic** (moving partial + sums/intermediate arrays across a host-mediated halo pass or a `!$omp declare target`-incompatible + branch), not diagnostics-guarded — do not conflate these with the diagnostics-transfer pattern. + +### 2.3 A fixed regression: `ff86497d5` "Fix small transfers in post_diabatic_halo_updates" (#174) + +Already merged into `dev/gpu`. Before the fix, `post_diabatic_halo_updates` (`MOM.F90:2105-2113`) +implicitly triggered a stream of small per-member transfers for `CS%tv%T`/`CS%tv%S` (accessed through +derived-type dereference inside a group-pass call) each time it ran. The fix wraps the call with an +explicit bulk map: +```fortran +! UMW NOTE: These transfers are needed to prevent excessive transfers in the group +! updates within this subroutine +!$omp target enter data map(to: CS%tv, CS%tv%T, CS%tv%S) +call post_diabatic_halo_updates(CS, G, GV, US, u, v, h, CS%tv) +!$omp target exit data map(from: CS%tv%T, CS%tv%S) +!$omp target exit data map(release: CS%tv) +``` +and removes a stale `! TODO: Safe? what about T and S?` comment next to the +`call do_group_pass(pass_uv_T_S_h, G%Domain, clock=id_clock_pass, omp_offload=.true.)` inside that +routine. This is a **derived-type deep-copy trap** (see `00-architecture.md` §2.3): touching +`CS%tv%T`/`CS%tv%S` through the `tv` derived type inside an `omp_offload` group pass was silently +causing the OpenMP runtime to materialize many small implicit transfers instead of one bulk one — +exactly the class of bug flagged generically in §7.5 ("implicit copy ... which cannot yet be +prevented"). + +--- + +## 3. In-flight: `diag_map_mediator_port` — what is actually being offloaded + +The branch touches the diag path in these commits (`git log dev/gpu..origin/diag_map_mediator_port +-- MOM_diag_mediator.F90 MOM_diag_remap.F90`): `b121aecbf` "separate" (+267/−73 across both files), +`fa796e4e0` "do concurrent the omp loops" (+82/−121 net, mostly loop-syntax cleanup on +`MOM_diag_mediator.F90` only), and `072db9355` "do concurrent the last loop I missed" (+5/−7, +`MOM_diag_remap.F90` only — a small follow-up converting one remaining plain loop to `do concurrent`). +**Verification note:** the *local* `diag_map_mediator_port` ref in this checkout has only the first +two of these (net `dev/gpu...diag_map_mediator_port` = +276/−121); `072db9355` lives only on +`origin/diag_map_mediator_port`, which is one commit ahead (net `dev/gpu...origin/diag_map_mediator_port` += +277/−124). Cite the remote ref when quoting the third commit. + +**This branch does *not* offload `send_data_infra`/FMS diag_manager writes, nor the top-level +`post_data`/`post_data_2d`/`post_data_3d` dispatch** — those remain plain host Fortran. What it +*does* offload, confirmed by `git diff dev/gpu...diag_map_mediator_port -- MOM_diag_mediator.F90 +MOM_diag_remap.F90`: + +1. **Mask setup** — `set_masks_for_axes` (`MOM_diag_mediator.F90:~800`, one-time init, not per + timestep): the 8 per-coordinate `mask3d` arrays (`mTL, mCuL, mCvL, mBL, mTi, mCui, mCvi, mBi`) are + now built with `do concurrent` directly on device-resident (`!$omp target enter data map(alloc: + ...)`) arrays instead of plain host loops, then pulled back with one bulk + `!$omp target exit data map(from: mTL, mCuL, mCvL, mBL, mTi, mCui, mCvi, mBi)` at the end. +2. **`diag_remap_calc_hmask`** (`MOM_diag_remap.F90:~496-550`) gained an explicit `h` argument (was + implicitly `remap_cs%h`) and is rewritten as two `do concurrent` kernels — a plain + `do concurrent(k, j, i)` zero-init and a `do concurrent(j, i) DO_LOCALITY(local(h_tot, h_err, k))` + vanished-layer mask loop with a serial inner `k` loop. (The intermediate commit `b121aecbf` + introduced these as `!$omp target teams distribute parallel do collapse(...)` kernels; the + follow-up `fa796e4e0` "do concurrent the omp loops" converted them, so the **net branch state is + `do concurrent`, not `omp target teams distribute`** — the whole branch mediator has **zero** + `target teams distribute` and 29 `do concurrent`.) The routine carries the doc comment *"Both mask + and h must already be present on the device (via prior enter data map)"* — i.e. it is now called + with device-resident arguments and does no transfer inside itself. +3. **Downsampling** — `downsample_field_2d/3d`, `downsample_mask_2d/3d`, and the driver + `downsample_diag_masks_set` are reworked to keep everything device-resident across the `dl=2, + MAX_DSAMP_LEV` loop, with new local pointer aliases (`m2dT, m3dTL, ...`) used specifically **"to + avoid derived-type deep-copy issues"** in `omp target` map clauses (the same trap as §2.3) — a + comment states explicitly: *"downsample_mask expects field_in on device"* / *"outputs stay on + device until the bulk map(from:) at the end of the c loop."* +4. **`post_data_2d_low`/`post_data_3d_low`** (`:1551-1582`, `:1892-1924`) — when a `conversion_factor` + forces a host-side copy into `locfield`, that copy is now explicitly pushed onto device + (`!$omp target enter data map(to: locfield)`) *before* `downsample_diag_field`/`downsample_field_*` + run, and explicitly removed (`!$omp target exit data map(delete: locfield)`) afterward — avoiding + an **implicit** map(to:)/map(from:) pair that the downsample kernels would otherwise trigger on + their own, one field at a time. + +> **Open (reviewed 2026-07-14):** does every caller on `diag_map_mediator_port` establish the device +> residency that the rewritten `diag_remap_calc_hmask`/`downsample_*` routines assume? The review could +> not settle this from source — it needs the specific caller-residency audit, and that audit is the +> gate before merging that branch: a caller handing in a host-only array would read uninitialized +> device memory silently. In particular, check the `h` argument threaded into `diag_remap_calc_hmask` +> is mapped at every call site, not just the mask. See KNOWLEDGE.md §9. + +### 3.1 Answer to Q3 — offload the mediator, or just cut transfers? + +**It is the latter, not the former.** The actual per-timestep hot path — `post_data_2d`/`post_data_3d` +dispatch, `post_data_2d_low`/`post_data_3d_low`'s masking/remap/`send_data_infra` sequence for the +*common* (no-conversion-factor, no-downsampling) case — is untouched. The branch instead targets the +**auxiliary, still-frequently-called machinery around** `post_data`: the one-time mask +setup/downsample-mask setup at init (§3.1/3.2 items 1–3, called once or per-restart, not per +timestep) and the **conversion+downsample side path inside `post_data_*_low`** (item 4, which only +fires for fields that have a non-1/0 unit conversion factor *and* are being downsampled). The goal +stated implicitly by the code comments throughout ("to avoid derived-type deep-copy issues", "expects +field_in on device", "keep field_in device-resident") is **transfer elimination for the +downsample/mask subsystem**, not turning the mediator into a GPU-resident diagnostics engine — the +FMS `diag_manager` write path (`send_data_infra`) fundamentally cannot be offloaded since FMS is +host-only in this fork, so a full "offload of the mediator" is not on the table; only its +device-adjacent pre-processing (masking, downsampling, unit conversion) is being made +transfer-free. + +### 3.2 Sibling branches: `feat/new-diag-manager`, `port-fms-diags` + +- `remotes/edoyango/port-fms-diags` diffs against `dev/gpu` at only **2 added lines** in + `MOM_diag_mediator.F90:1122`: `!$omp target enter data map(to: axes, axes%mask3d)` at the end of + `define_axes_group` — an exploratory first step toward keeping axis masks device-resident, clearly + a precursor idea to (and superseded by) `diag_map_mediator_port`'s more complete mask handling. +- `remotes/edoyango/feat/new-diag-manager` is mostly build-system work (FMS submodule/yaml + `configure.ac` support) plus one directly relevant commit, `b2a30750a` "Step_MOM_tracer_dyn: Remove + data transfers" (not yet on `dev/gpu`): it **deletes** the blanket + `!$omp target update from(h, CS%uhtr, CS%vhtr)` / `to(...)` pair that used to unconditionally + bracket every call to `step_MOM_tracer_dyn` in `MOM.F90:996-1002`, and **relocates** the transfers + *inside* `step_MOM_tracer_dyn` gated behind the actual conditions that need host data: + `CS%debug` (checksums), `CS%use_particles .and. CS%use_uh_particles` (Lagrangian particle + tracking, host-only), `associated(CS%OBC)` (open boundary tracer reservoirs), and + `allocated(CS%tv%SpV_avg)` (derived thermodynamics). It also converts `CS%uhtr(:,:,:) = 0.0` / + `CS%vhtr(:,:,:) = 0.0` resets into `do concurrent` device kernels. This is the same "push the + transfer down to the actual conditional consumer" strategy as §2.1/§2.2, applied one call frame + deeper than where `dev/gpu` currently does it — a preview of where the blanket + `MOM.F90:1091`-style transfers are headed. + +--- + +## 4. Remaining host-only regions in the timestep and the transfers they force (Q4) + +Per `00-architecture.md` §6.3, the following are **untouched on `dev/gpu`** (diff line counts against +`dev-gfdl`): `MOM_diabatic_driver.F90` (3 lines), `MOM_set_diffusivity.F90` (0), +`MOM_CVMix_KPP.F90` (0), `MOM_energetic_PBL.F90` (0), `MOM_mixed_layer_restrat.F90` (0), +`MOM_ALE.F90` (3), `MOM_regridding.F90` (0), `MOM_remapping.F90` (0), +`MOM_diag_mediator.F90` (0), and (confirmed here) `MOM_restart.F90` (0). + +Each host-only region is bracketed in `MOM.F90`/`step_MOM_thermo` by an explicit +`target update from(...)` / `target update to(...)` pair so the device-resident state stays +consistent around the host detour: + +| Host-only region | Bracketing transfer (in) | Bracketing transfer (out) | +|---|---|---| +| `diabatic`/`layered_diabatic` (`MOM_diabatic_driver.F90`) | implicit — dycore state already host-visible via surrounding brackets | `MOM.F90:1043` explicit `map(to: CS%tv, CS%tv%T, CS%tv%S)` before `post_diabatic_halo_updates` | +| ALE regrid/remap (`ALE_regridding_and_remapping` → `MOM_ALE.F90`/`MOM_regridding.F90`/`MOM_remapping.F90`) | `MOM.F90:1036` `!$omp target update from(u, v, h)` | `MOM.F90:1038` `!$omp target update to(u, v, h)` | +| `write_energy` — T/S consumption (`MOM_sum_output.F90:762`) | `!$omp target update to(tv%S, tv%T)` (host-modified-by-diabatic T/S pushed back to device) | n/a (device-resident kernel follows) | +| `find_eta`/diagnostics needing `z_top`,`Z_0APE` (`MOM_diagnostics.F90:968`, `MOM_sum_output.F90:705`) | `target update to(h)` / `to(Z_0APE)` | `target exit data map(from: z_top)` | +| MEKE / thickness-diffuse / lateral mixing coeffs | `MOM_lateral_mixing_coeffs.F90:273` (`if (CS%calculate_cg1)`) and `:300` (`if (CS%BS_use_sqg_struct .or. ... .or. CS%id_sqg_struct>0)`), each `!$omp target update from(h)` before the host-only `wave_speed`/`calc_sqg_struct` | none in this file — `h` is pulled read-only, never pushed back | +| `calculate_diagnostic_fields` fan-out (all `id_xxx` diagnostics) | `MOM.F90:1091` blanket `target update from(u, v, h, CS%uhtr, CS%vhtr)` | none (read-only consumption) | +| Restart write/read (`MOM_restart.F90`, all of it) | **entirely implicit** — see §6 | **entirely implicit** | + +**MEKE** (`MOM_MEKE.F90`) is not in the untouched list in §6.3 of the architecture doc but is +mentioned as a target-of-interest here; it was not found to have its own `target update from` +sites distinguishing it from the general dycore pattern — its interaction with diagnostics/host +regions is via the same `CS%visc`/`CDp` shared containers audited above (`MOM.F90:1380-1381`, +`1807-1812` guard `CS%visc%Ray_u/v`, `bbl_thick_u/v`, `Kv_bbl_u/v` — all `if (allocated(...))` +guarded rather than diagnostic-id guarded, since these are set-viscosity/BBL outputs, not +attached to a specific diagnostic id): +```fortran +!$omp target update from(CS%visc%Ray_u) if (allocated(CS%visc%Ray_u)) +!$omp target update from(CS%visc%Ray_v) if (allocated(CS%visc%Ray_v)) +!$omp target update from(CS%visc%bbl_thick_u) if (allocated(CS%visc%bbl_thick_u)) +!$omp target update from(CS%visc%bbl_thick_v) if (allocated(CS%visc%bbl_thick_v)) +!$omp target update from(CS%visc%Kv_bbl_u) if (allocated(CS%visc%Kv_bbl_u)) +!$omp target update from(CS%visc%Kv_bbl_v) if (allocated(CS%visc%Kv_bbl_v)) +``` +(`MOM.F90:1807-1812`) — an `allocated()`-guarded variant of the same "only transfer if it's actually +going to be used" idiom, one level removed from a diagnostic id (these fields are allocated only +when the corresponding BBL/viscosity option is active, so the guard is a config-time rather than a +diagnostic-active-time gate). + +--- + +## 5. Profiling: nvtx markers piggybacking on `cpu_clock` (Q5) + +Branch `remotes/edoyango/benchmark_ALE_nvtx_clocks`, commit `ae67665d3` "add nvtx markers to clocks" +(+13/−0 twice, identical patch to both `config_src/infra/FMS1/MOM_cpu_clock_infra.F90` and +`config_src/infra/FMS2/MOM_cpu_clock_infra.F90`): + +```fortran +use nvtx +... +integer, parameter :: MAX_NVTX_CLOCKS = 4096 +character(len=64), save :: nvtx_clock_names(MAX_NVTX_CLOCKS) = "" +... +subroutine cpu_clock_begin(id) + integer, intent(in) :: id + if (id > 0 .and. id <= MAX_NVTX_CLOCKS) then + if (len_trim(nvtx_clock_names(id)) > 0) call nvtxStartRange(trim(nvtx_clock_names(id))) + endif + call mpp_clock_begin(id) +end subroutine cpu_clock_begin + +subroutine cpu_clock_end(id) + integer, intent(in) :: id + call mpp_clock_end(id) + if (id > 0 .and. id <= MAX_NVTX_CLOCKS) then + if (len_trim(nvtx_clock_names(id)) > 0) call nvtxEndRange + endif +end subroutine cpu_clock_end + +integer function cpu_clock_id(name, sync, grain) + ... + cpu_clock_id = mpp_clock_id(name, flags=clock_flags, grain=grain) + if (cpu_clock_id > 0 .and. cpu_clock_id <= MAX_NVTX_CLOCKS) then + nvtx_clock_names(cpu_clock_id) = name + endif +end function cpu_clock_id +``` + +**Mechanism:** MOM6 already instruments essentially every named phase of the timestep with +`cpu_clock_id("name")` at init + `cpu_clock_begin`/`cpu_clock_end` pairs around the corresponding +code (this predates the GPU port — it is the pre-existing FMS `mpp_clock` profiling infra used for +the text-based clock summary at the end of a run). This commit **transparently wraps** that existing +infra: every clock name registered via `cpu_clock_id` is cached, and every subsequent +`begin`/`end` call also opens/closes an NVTX range of the same name — **with zero changes anywhere +else in the codebase**. This means `id_clock_diagnostics`, `id_clock_diag_mediator`, +`id_clock_pass`, `id_clock_thermo`, `id_clock_tracer`, `id_clock_dynamics`, `id_clock_other`, etc. +(the clocks bracketing exactly the diagnostics/IO regions audited in §1–§4) automatically show up +as named regions in an Nsight Systems (`nsys`) timeline, without writing any new instrumentation. + +**What this reveals:** because the granularity is whatever the *existing* CPU-profiling clock +hierarchy already had, the nvtx timeline directly exposes (a) how much wall time +`id_clock_diagnostics`/`id_clock_diag_mediator` consume relative to the dycore clocks, and (b) via +the visual gaps/overlaps in the Nsight timeline, where a `target update from`/`to` pair (§2, §3, §4) +stalls the GPU stream waiting on a host-side diagnostics or restart region. It reuses the *pre-existing +naming convention* rather than requiring an nvtx-specific taxonomy, so anyone already familiar with +MOM6's clock summary output can read the nvtx timeline without relearning names. + +`remotes/edoyango/devgpu-w-viscmlclock`, commit `5bbf92a3a` "add tmp clock" (+7 lines, +`MOM_set_viscosity.F90` only) is a much smaller, single-file exploratory addition of one extra +`cpu_clock_id`/`begin`/`end` pair around (implied by the branch name) the mixed-layer viscosity +computation — a manual instrumentation add-on rather than an infra change, presumably to get +finer-grained nvtx visibility into `set_viscous_ML` specifically once `ae67665d3`'s wrapper is in +place upstream of it. + +--- + +## 6. Restart IO: fully host-staged (Q6) + +`src/framework/MOM_restart.F90` has **zero** diff against `dev-gfdl` and **zero** OpenMP directives +of any kind (`grep -c "target update\|omp target\|do concurrent" MOM_restart.F90` → 0/0/0 across all +three patterns). The restart registry (`register_restart_field`, `save_restart`, `restore_state`) is +untouched: it operates purely on whatever host-resident copy of a field it is handed. This means +**every restart write requires the relevant device arrays to already have been pulled to host** by +whatever caller invoked the restart save — restart IO does not do its own transfer, it inherits +host-valid data from the surrounding `target update from` brackets already present in `MOM.F90` +(e.g. the same blanket `u,v,h,uhtr,vhtr` transfers at synchronization points, §2.2, cover most of +what a restart file needs) or, if none happens to be in scope at the restart-write call site, would +silently write stale host memory — a latent correctness risk worth flagging for anyone porting a +new CS member that also participates in the restart registry (`MOM_variables.F90:294` comment on +pointer members being restart-registry targets, cross-ref `00-architecture.md` §2.1). No branch in +this repo currently makes any part of the restart path device-aware; it is treated purely as a +serial, host-side bookkeeping concern (consistent with `.testing/tools/track_gpu_port.py`'s +`!@start noport` sentinel category described in `00-architecture.md` §8, though `MOM_restart.F90` +itself carries no such sentinels — it simply has nothing device-related to mark). + +> **Resolved (2026-07-14):** the stale-host risk is **latent, not live**. `save_MOM_restart` does no +> transfer of its own, but `step_MOM`'s sync-point blanket `update from(u, v, h, CS%uhtr, CS%vhtr)` +> runs whenever `MOM_state_is_synchronized(CS)` — the same condition under which the driver writes +> restarts — and the thermo/mixing fields are host-authoritative because diabatic is host-only. The +> standing rule: any *newly* device-resident restart-registered field must be added to a dominating +> `update from` before the restart save. + +--- + +## 7. Summary table — file status at a glance + +| File | Diff vs `dev-gfdl` | Directives | Status | +|---|---|---|---| +| `MOM_diag_mediator.F90` | 0 (unchanged) | 0 | Fully host; `diag_map_mediator_port` in flight (§3) | +| `MOM_diag_remap.F90` | 0 on `dev/gpu` (changed only on `diag_map_mediator_port`) | 0 on `dev/gpu` | Same | +| `MOM_restart.F90` | 0 (unchanged) | 0 | Fully host, no in-flight branch | +| `MOM_diagnostics.F90` | +7/−0 | 3 guarded `target update from` | Minimal, transfer-minimization only | +| `MOM_sum_output.F90` (`write_energy`) | +34/−22 (56 lines touched) | multiple `do concurrent` + 2 `target update to` (`:705` `Z_0APE`, `:762` `tv%S,tv%T`) | **Compute is GPU-resident**; bridges `to(tv%S,tv%T)`/`to(Z_0APE)` from host-modified-by-diabatic inputs | +| `MOM.F90` (driver-level brackets) | part of +260/−40 | 56 `target update from`, ~6 guarded | Mostly unconditional blanket transfers around host-only regions | + +--- + +## 8. Prescriptive rules for a porting agent (distilled) + +When you port a module that computes device-resident fields and *also* posts diagnostics or feeds +the restart registry, apply these rules. They encode the trade-off the codebase has actually made: +**blanket `target update from` at coarse synchronization points, not per-field guards everywhere.** + +### 8.1 Handling `post_data` calls in a newly ported module + +1. **Never assume `post_data_*` sees device data.** `MOM_diag_mediator.F90` has **zero** OpenMP + directives and is unchanged on `dev/gpu` (§1). Any array you pass to `post_data` must already be + **host-valid**. The transfer is *your* responsibility as the producer, not the mediator's. +2. **Decouple the transfer from the post.** Do **one** `!$omp target update from()` covering + *all* diagnostics that consume those fields, then keep the individual `if (CS%id_xxx > 0) call + post_data(...)` guards. Do not put a separate `update from` inside each `if (id>0)`. The canonical + shape is `MOM_tracer_hor_diff.F90:719-733` (§2.1): a single guarded transfer covering both the + `CS%debug` checksum path and both diag ids, followed by two independent `post_data` guards. +3. **Collapse multiple ids into one gate** with `any([...] > 0)` when several diagnostics derive from + the same raw field — see `MOM_diagnostics.F90:1825-1827` (`uhtr`/`vhtr`/`h` each gate several + downstream ids). This is strictly better than N separate transfers of the same array. +4. **Where to place the transfer:** immediately before the *first* consumer, at the widest scope + where the field is still known host-valid. For a whole fan-out of unrelated diagnostics (the + `calculate_diagnostic_fields` case) the codebase deliberately does **one blanket** transfer at the + driver level (`MOM.F90:1091`, `target update from(u, v, h, CS%uhtr, CS%vhtr)`) rather than gating + each of dozens of `if (id>0)` posts — simpler to reason about, at the cost of always paying the + transfer at every synchronization boundary. **This blanket-at-sync-points choice is the codebase + default; match it unless you have a measured reason to push guards deeper.** +5. **Know the guarded ideal and where it's headed.** The finer-grained alternative — relocate each + transfer down to the *actual conditional consumer* (`CS%debug`, `associated(CS%OBC)`, + `allocated(CS%tv%SpV_avg)`, a specific `id`) — is previewed on `feat/new-diag-manager` + (`b2a30750a`, §3.2) and is the direction of travel, but is **not** what merged `dev/gpu` does at + the `MOM.F90:1091` frame today. Prefer the blanket form for new work at sync points; use guarded + push-down only when profiling (§5) shows the blanket transfer is a real stall. + +### 8.2 Restart-registered arrays a module mutates on device + +1. If your module `register_restart_field`s an array **and** mutates it inside a `target` region, + that array is device-resident between mutation and the next restart write. `MOM_restart.F90` does + **no** transfers of its own (§6) — it writes whatever host memory it is handed. +2. Ensure a `target update from()` dominates every `save_restart` call site that will write + it. In practice the §2.2 blanket transfers at synchronization points already cover the core + prognostic set; a **new** restart-registered device array outside that set (especially a + pointer-member restart target, `MOM_variables.F90:294`/§2.1) needs its own transfer or it will be + written stale. This is the concrete latent trap flagged in §6. +3. Do **not** try to make `MOM_restart.F90` device-aware — no branch does, and it is treated as + serial host bookkeeping (the `!@start noport` category, `00-architecture.md` §8). + +### 8.3 Checking you haven't left a stale-host trap + +- **Grep discipline:** for every `post_data`/`save_restart`/host-only `call` in your ported module, + confirm a `target update from` for its argument fields exists on every path reaching it. The + transfer must be *upstream* (dominate) the consumer, not merely present in the file. +- **Bracket host-only detours:** any host-only region you cannot avoid (ALE remap, diabatic, energy + sums) must be wrapped `from(...)` before / `to(...)` after so device state is restored — pattern + `MOM.F90:1036/1038` (ALE) and `MOM_sum_output.F90:762` (`to(tv%S,tv%T)` pushing diabatic-modified + host T/S back up). Direction matters: `from` when the host will *read* device data, `to` when the + host has *modified* data the device must see next. +- **Verify with checksums, not eyeballs:** a stale-host transfer bug is invisible unless a + diagnostic/restart field diverges. Compare `MOM_checksums` hchksum/uchksum and reproducing-sum + energy output CPU vs GPU (`00-architecture.md` §7.2, §9) — a stale field shifts the bitcount + checksum. `CS%debug`-gated `uvchksum` calls (e.g. `MOM_tracer_hor_diff.F90:725`) exist precisely to + catch this class. +- **Watch the derived-type deep-copy trap:** touching `CS%tv%T`/`CS%tv%S` (or any derived-type member + array) inside an `omp_offload` group pass or a `map` clause silently materializes many small + implicit transfers instead of one bulk transfer (§2.3, `ff86497d5`). Wrap such calls in an explicit + `map(to: CS%tv, CS%tv%T, CS%tv%S)` / `map(from: ...)` bracket, or alias the member to a bare pointer + first (the `diag_map_mediator_port` `mTL`/`m2dT` pattern, §3). + +## 9. Cross-references + +- `00-architecture.md` §2.3 (derived-type deep-copy cost — the same trap fixed in `ff86497d5` and + worked around throughout `diag_map_mediator_port`'s pointer-alias pattern), §7.4, §8 (port-coverage + tooling / `!@start noport` sentinels, relevant to why `MOM_restart.F90` has none). +- `03-openmp-mapping.md` for the general enter-data/exit-data lifecycle idiom reused throughout §2–§3 + here. + +--- + +## Verification notes + +Verified against source + git on branch `dev/gpu` (baseline `dev-gfdl`); no code built or run. + +**Confirmed exactly (checked against code/git):** +- `MOM_diag_mediator.F90` and `MOM_restart.F90` empty `dev-gfdl...dev/gpu` diffs; `MOM_restart.F90` + 0/0/0 for `target update`/`omp target`/`do concurrent`; `MOM_diag_remap.F90` empty on `dev/gpu`. +- Total `omp target update from` across `src/` = **249**, and every per-file count in §2 (MOM.F90 56, + MOM_barotropic 52, MOM_hor_visc 53, dyn_split_RK2 45, CoriolisAdv 11, set_viscosity 7, + tracer_hor_diff 6, PressureForce_FV 5, vert_friction 4, interface_heights 3, diagnostics 3, + lateral_mixing_coeffs 2, tracer_advect 1, state_initialization 1) — all exact. +- All guarded-transfer sites and their guard conditions: `MOM_tracer_hor_diff.F90:722`, + `MOM_diagnostics.F90:1825/1826/1827`, `MOM_PressureForce_FV.F90:1301`. Blanket sites `MOM.F90:1091`, + the commented-out dead duplicate near `:1113`, ALE brackets `MOM.F90:1036/1038`. `allocated()`-guarded + visc transfers `MOM.F90:1807-1812`; `lateral_mixing_coeffs.F90:273/300`. +- Commit `ff86497d5` (#174, author uwagura): reconstructed from the diff — 5 insertions/1 deletion, + adds `map(to: CS%tv, CS%tv%T, CS%tv%S)` before / `map(from:)` + `map(release:)` after the + `post_diabatic_halo_updates` call (`MOM.F90:1042-1046`) and removes the `! TODO: Safe? what about T + and S?` comment at the group pass. Derived-type deep-copy characterization is sound. +- `write_energy` transfers: `MOM_sum_output.F90:705` `to(Z_0APE)`, `:762` `to(tv%S, tv%T)` — direction + and host-diabatic-mutation rationale confirmed. +- `diag_map_mediator_port` content: pointer-alias mask setup (`mTL/mCuL/mCvL/mBL`, `m2dT/m3dTL/...`), + `diag_remap_calc_hmask` gained explicit `h` arg + "must already be present on the device" comment, + `downsample_diag_masks_set` rework, `locfield` `map(to:)`/`map(delete:)` around the + conversion+downsample side path in `post_data_2d_low`/`post_data_3d_low`. Branch does **not** touch + `send_data_infra`/FMS or the top-level `post_data` dispatch (Q3 answer "cut transfers, not offload + the mediator" is correct). Per-commit stats `b121aecbf` +267/−73, `fa796e4e0` +82/−121, + `072db9355` +5/−7 all match. +- Sibling branches: `port-fms-diags` = 2 added lines (`map(to: axes, axes%mask3d)` at + `define_axes_group`); `feat/new-diag-manager` `b2a30750a` touches MOM.F90 + diagnostics + + tracer_advect + tracer_hor_diff (consistent with the "push transfers to conditional consumer" + description). nvtx commit `ae67665d3` +13/+13 to both FMS1/FMS2 `MOM_cpu_clock_infra.F90` — quoted + code (MAX_NVTX_CLOCKS=4096, nvtxStartRange/EndRange wrapping) matches verbatim. `5bbf92a3a` +7 lines, + `MOM_set_viscosity.F90` only. +- All §1 call-map line numbers (post_data generic `:74`, `post_data_2d :1408`, `_2d_low :1436`, + `post_data_3d :1585`, `_3d_low :1750`, `by_column :1927`, `by_point :1945`, + `diag_update_remap_grids :3655-3759`, `diag_copy_diag_to_storage :4130-4146`, + `diag_copy_storage_to_diag :4149-4164`) and the `calculate_diagnostic_fields` `id_u/id_v/id_h/id_usq` + guards at `MOM_diagnostics.F90:305-320`. Summary-table diffstats `MOM_diagnostics +7/−0`, + `MOM.F90 +260/−40` confirmed. + +**Corrected:** +1. **§3 branch/commit provenance:** the *local* `diag_map_mediator_port` ref here has only 2 of the 3 + commits; `072db9355` exists only on `origin/diag_map_mediator_port` (remote is one commit ahead). + Doc now qualifies the ref and gives both net diffstats (local +276/−121, remote +277/−124). +2. **§3 item 2 `diag_remap_calc_hmask` kernel form:** the *net* branch state uses two **`do concurrent`** + kernels, not `!$omp target teams distribute parallel do collapse(...)`. The omp-teams form existed + only in the intermediate commit `b121aecbf`; `fa796e4e0` converted it (branch mediator: 0 + `target teams distribute`, 29 `do concurrent`). Corrected in place with the history noted. +3. **§7 summary table `MOM_sum_output.F90`:** was `+56/−22`; the true diffstat is **+34/−22** (56 lines + *touched*). Corrected. + +**Confidence:** High. Every load-bearing factual claim (empty diffs, the 249 count and all per-file +counts, the five guarded sites and their conditions, `ff86497d5`, the `write_energy` transfers, the +`diag_map_mediator_port` scope and its four offload items, the nvtx wrapper code, all §1 line numbers) +was verified directly against source or git and matches. Of the two items previously raised for +review: (a) the restart "stale host" risk is settled — latent, not live (§6); (b) whether the +`diag_map_mediator_port` device-residency contract is honored by all callers remains open (§3). diff --git a/knowledge/gpu-knowledge/13-compiler-workarounds.md b/knowledge/gpu-knowledge/13-compiler-workarounds.md new file mode 100644 index 0000000..2464627 --- /dev/null +++ b/knowledge/gpu-knowledge/13-compiler-workarounds.md @@ -0,0 +1,332 @@ +# Compiler Workarounds — the `dev/gpu` nvfortran/NVHPC Bug Catalogue + +> **Purpose.** This is the grep-friendly master index of every nvfortran/NVHPC compiler bug, +> workaround, and directive hit while porting MOM6 to NVIDIA GPUs on `dev/gpu`. Every row below is +> anchored to a `file:line` or a commit hash so it can be verified with `git show ` or +> `sed -n 'p' `. Read `00-architecture.md` §7.5 first. Deeper narrative treatments of +> some of these already exist in `04-do-concurrent-patterns.md` §3–4, `06-eos-layer.md`, and +> `08-cross-module-inlining.md` §5 — this document is the one-stop index that also adds items those +> docs don't cover (the A100/25.5 crash, struct-of-arrays flattening, in-flight private-clause bugs, +> the OpenACC→OpenMP migration history, and the `__NVCOMPILER_OPENMP_GPU` macro). +> +> **Method note.** No build or run was performed to produce this document; every entry is derived +> from source comments, directives, and `git log`/`git show` on `dev-gfdl..dev/gpu` plus the +> `origin/*` side branches. "Still open?" reflects what the comment/commit says, not independent +> verification. + +--- + +## 1. Quick-index table + +| # | Symptom / bug | Anchor | Workaround | Category | Still open? | +|---|---|---|---|---|---| +| 1 | Polymorphic `this` dispatch causes **runtime errors on GPU** in `do concurrent` | `MOM_EOS_Roquet_rho.F90:261,334,449,465,557,780,852,891`; `MOM_EOS_Wright.F90:107-109,230`; commits `7c7af5572`, `52a1b3954` | Duplicate each `elemental` type-bound proc as a free `_loc` function taking no `this`; array wrappers call `_loc` inside `do concurrent` | Genuine compiler bug (v-table/device dispatch) | **Fixed** for Roquet_rho & buggy_Wright; **all other EOS forms** (linear, UNESCO, Jackett06, TEOS10, Wright_full/red, Roquet_SpV) still use polymorphic elemental dispatch and are **not** GPU-safe | +| 2 | Residual **implicit device copy of `this`**. Verbatim comment (all 5 sites): *"NOTE: There is an implicit copy of \`this\` which cannot yet be prevented."* (only `Wright:1008,1048` add: *"Possibly because Nvidia cannot associate \`this\` with \`EOS%type\`."*) | `MOM_EOS_Wright.F90:1008,1048,1114,1147`; `MOM_EOS_Roquet_rho.F90:817` | None — noted, not fixed | Genuine compiler bug (suspected) | **Open / unresolved** | +| 3 | `do concurrent` **cannot reduce into an array element** (`itmp(i,j)`-style reduction target) | `MOM_tracer_hor_diff.F90:962` | Reduce into a scalar (`itmp`) with `DO_LOCALITY(reduce(max:itmp))`, then assign scalar into the array after the loop | Genuine compiler bug / OpenMP-standard restriction (reduction var must be a named scalar) | Worked around; also documented in `04-do-concurrent-patterns.md` §4.4 | +| 4 | `modulo()` **not implemented on all systems** (incl. NVIDIA GPU device runtime) | `MOM_intrinsic_functions.F90:232`; upstream fix commit `9aea28954` (branch `cuberoot-no-modulo`, merged) | Replace `modulo(e,3)` with explicit `e_a - e_r*3` arithmetic in `rescale_cbrt` | Genuine compiler/library gap | **Fixed**, merged into current source | +| 5 | `x**(1.0/n)` lowers to `exp((1/n)*log(x))` — **imprecise / bit-different** vs. host libm | `MOM_intrinsic_functions.F90:120-131` (`nth_root`, doc comment) | Fixed-iteration Newton's method on `y^n - x = 0` using only `*,+,/`; used by `MOM_barotropic.F90`'s `bt_rem = av_rem**Instep` | Numerical-portability workaround (not a bug, a bit-repro requirement) | Resolved by design | +| 6 | `do concurrent` on the **meridional tracer-flux face loop gives the wrong result** | `MOM_tracer_hor_diff.F90:1464` — `! this gives wrong result when using do concurrent on NVHPC 25.9` | Kept as `!$omp target teams loop collapse(2) private(...)` instead of `do concurrent` | **Genuine compiler bug, version-specific (NVHPC 25.9)** | Open (workaround in place; not re-tested on later NVHPC) | +| 7 | **Early `exit` inside a `do concurrent`-nested loop gives wrong answers** | Commit `e23d6a7b1` — *"NVHPC 25.11 didn't like the early exit and would give wrong answers."* `MOM_tracer_hor_diff.F90` insertion-sort loop | Replace `do k2=k,2,-1 ; if (cond) exit` with an `if`-guarded loop body (`if (cond) then ... endif ; enddo`, no `exit`) | **Genuine compiler bug, version-specific (NVHPC 25.11)** | Fixed by rewrite; pattern (avoid `exit`/`return`/`cycle` inside device-offloaded loop bodies) should be treated as a hard rule | +| 8 | OpenMP runtime **under-launches teams** (17 vs. ~238 expected) for a target-teams region | Commit `5b5f6b2b1` — *"For some reason omp runtime was only starting a kernel with 17 blocks when the openacc version would start it with 238 or something like that."* `MOM_continuity_PPM.F90:701,707` (`nteams = ceiling(...) / 128.`) | Manually compute `num_teams(nteams)` from the tile's iteration count instead of relying on the runtime's default team count | Genuine compiler/runtime scheduling deficiency | **Open / permanent workaround** — the manual `num_teams` clause is load-bearing, not decorative | +| 9 | Mandatory function inlining or **wrong answers** | Commit `3cb184edd` — *"IMPORTANT: However for OpenMP, inlining of ratio_max and flux_elem is MANDATORY. do so with `-Minline=name:ratio_max,name:flux_elem`. Otherwise results are incorrect."* | `-Minline=name:...` compiler flag (later superseded, see #10) | Genuine compiler bug (cross-procedure device call miscompiled without inlining) | Superseded, not re-verified as fixed | +| 10 | Avoiding the fragile `-Minline=name:...` flag list | Commit `4e3f1b758` — *"add !NVF\$ INLINE to ratio_max flux_elem … instead of compiling with -Minline=name:flux_elem,name:flux_elem_OBC,name:ratio_max, can compile with -Minline=pragma instead."* | Source-level `!NVF$ INLINE` directive + `-Minline=pragma` build flag | Build-portability improvement | Superseded again by #11 | +| 11 | `!NVF$ INLINE` replaced with a portable directive; **large -O2 perf win** | Commit `93dbbd36e` — *"remove nvf inline and replace with intel forceinline … Significantly improves performance of blocked zonal/meridional_mass_flux at -O2"* | `!DIR$ ATTRIBUTES FORCEINLINE :: flux_elem` / `flux_elem_OBC` (`MOM_continuity_PPM.F90:1086,1149`); `ratio_max` (`:3086`) now has **no** inline directive at all (relies on `pure function` + compiler default) | Performance-driven directive swap | Current state; **`-Minline`/forceinline is still effectively load-bearing for correctness per #9**, so this is a correctness-adjacent perf change, not a pure perf change | +| 12 | `thread_limit(128)` removed from the same `target teams` region that got `num_teams` | Commit `4e3f1b758` diff: `-!$omp target teams num_teams(nteams) thread_limit(128)` → `+!$omp target teams num_teams(nteams)` (`MOM_continuity_PPM.F90:707`) | Dropped `thread_limit` clause; `num_teams` alone retained | Perf/behavior tuning | Not explained in commit message — flagged for follow-up | +| 13 | `omp target teams loop` region **crashed / segfaulted** | Commit `5274c3a8e` "fix segfault" — removed `!$omp target` / `!$omp parallel loop collapse(2)` wrapper around a k-recurrence loop in `PressureForce_FV_Bouss`, replaced with `do concurrent` | `do concurrent (j=js:je, I=Isq:Ieq)` in place of `!$omp target … !$omp parallel loop collapse(2) … !$omp end target` | Genuine compiler/runtime bug (crash) | Fixed by rewrite | +| 14 | General instability of `!$omp target` regions on some compilers | Commit `0f05b360f` — *"PGF: Convert omp target region to do concurrents. Some compilers do not handle the omp target syntax very well. Switching to do concurrent seems to minimize these issues."* (Note: "PGF" here = the **Pressure Gradient Force** module prefix used throughout `MOM_PressureForce_*.F90`/`MOM.F90` commit messages, **not** the PGI/PGF90 compiler.) | Prefer `do concurrent` over `!$omp target`/`!$omp parallel loop` where both are equally expressive | General reliability preference, feeds guiding principle §0.3 of `00-architecture.md` | Ongoing house rule | +| 15 | Reversion: `omp target teams loop` → `do concurrent` | Commit `e8b0ecfbf` "omp target teams loop -> do concurrent" (`MOM_continuity_PPM.F90`, −85/+54 lines) | Same direction as #14 | Reliability/perf preference | Merged | +| 16 | **A100 (Stellar) crash with nvfortran 25.5** | Commit `2108e0eba` — *"Remove eta_bt transfer from find_eta_2d that was causing crashes on stellar A100s with nvfortran 25.5"* — removed `!$omp target enter data map(to: eta_bt) if (present(eta_bt))` from `find_eta_2d` (`MOM_interface_heights.F90`) | Delete the conditional `map(to:)` of an **optional** dummy argument guarded by `if (present(...))`; rely on `eta_bt` being resident via its caller's own mapping | **Genuine compiler bug, version+arch specific (NVHPC 25.5, A100)** | Fixed by removal; the underlying pattern (`map(...) if (present(optional_arg))`) is suspect in general — see #17 | +| 17 | Conditional `!$omp target enter data if(...) map(to:...)` lines commented out during the OpenACC→OpenMP migration (**see corrected history below — the original "four `present()` lines, never enabled" claim was wrong**) | Commit `f74525ae8` "Transition OpenACC to OpenMP" — *"We lose present() but overall it seems to work."* added **four** `!!!$omp target enter data if(...) &` commented blocks to `MOM_PressureForce_FV.F90`, of which **only one** uses the Fortran `present()` intrinsic (`if(present(pbce))`); the other three are `if(use_EOS)`, `if(use_p_atm)`, `if(.not. use_p_atm)` | The `if(present(pbce))` block was **un-commented and enabled the same day** by `d4ba8d69d` "OpenMP: PBCE on GPU" (2024-12-06), then **refactored out** by `9bfe7d358` "PF: Move pbce and eta management out of fn" (2025-01-14). **None of the four lines exist in current source.** | The "We lose present()" quote refers to the Fortran **`present(optional_arg)` intrinsic**: the diff carries no OpenACC `present()` data clauses at all, only commented-out `if(present(pbce)) map(to: pbce)` conditional maps | **Not open in the way originally stated.** The `present(pbce)+map` pattern *was* enabled and used for ~5 weeks, then removed by a **refactor**, not because of a compiler crash. Its link to bug #16 (an actual A100/25.5 crash on `map(to:eta_bt) if(present(eta_bt))`) is a foreshadowing of the identical construct, not causal evidence — see §2.1 #17 | +| 18 | Struct-of-arrays member arrays are expensive to **attach/detach** on device | Commit `1865612de` "Convert structs of arrays to flat arrays" — *"Flattening these arrays halves time when compiling for GPU. Lots of time was being spent 'attaching' and 'detaching' the member arrays to/from each struct on the GPU."* `MOM_tracer_hor_diff.F90` (−155/+131 lines) | Flatten arrays-of-derived-type-members into plain flat arrays indexed by a combined index, at the cost of ~20% more memory (700k→830k elements/array in benchmark) | Performance workaround for deep-copy/attach overhead (not a bug) | Merged; documented also in `00-architecture.md` §2.3 and `01-memory-control-structures.md` | +| 19 | `do concurrent` locality specifiers (`local`, `reduce`) **not supported on all compilers in active use** | Commits `d2a72eddd` "DO_LOCALITY compatibility macro", `cd178dd52` "DO_LOCALITY() bugfix" | `DO_LOCALITY(X)` macro in `src/framework/do_concurrent_compat.h`: expands to `X` if `HAVE_FC_DO_CONCURRENT_LOCAL` else to `;` (a no-op that avoids a dangling line-continuation `&` parse error). Feature-detected by `ac/m4/mom6_fc_do_concurrent_local.m4` → `ac/configure.ac:172` → `HAVE_FC_DO_CONCURRENT_LOCAL` | Portability/feature-detection (not nvfortran-specific; guards **against compilers that don't have it**, e.g. older gfortran) | Resolved by design; permanent infra | +| 20 | `do concurrent` formatting broke some compilers' parsers | Commit `e5444b4e5` "expand and indent do concurrents for gcc" | Reformat compact `do concurrent (...) ; stmt ; enddo` one-liners into expanded/indented multi-line form | Cross-compiler portability (gcc/gfortran, not nvfortran) | Merged | +| 21 | `makedep` (the in-house Fortran dependency scanner) **couldn't parse empty macro functions** or valueless `-D` flags, needed for `HAVE_FC_DO_CONCURRENT_LOCAL`-style defines | Commit `6474597b1` "Makedep: Support empty macro functions" | Parser patch: supports `#define foo(x)` with no body, and command-line `-DMACRO` without `=value` | Build-tooling fix, prerequisite for #19's infra | Merged | +| 22 | `__NVCOMPILER_OPENMP_GPU` macro used to change **default block sizes and disable CPU-only early-exit optimizations** | `MOM_continuity_PPM.F90:1406,1434,1442,2416,2443,2450,3120`; `MOM_CoriolisAdv.F90:2094`; commit `93dbbd36e` body: *"if `__NVCOMPILER_OPENMP_GPU` macro is defined (to be replaced at a later time), set default n?block to 0"* | `#ifdef __NVCOMPILER_OPENMP_GPU` / `#ifndef` guards: (a) default `niblock/njblock/nkblock = 0` (whole-domain, GPU) vs. `32/4/1` (CPU); (b) disable the `domore`/`if (.not.domore) exit` early-exit convergence check entirely under GPU builds — GPU kernels always run the fixed iteration count instead of testing for early convergence | Both a compile-time tuning knob (block sizes) **and** an early-exit avoidance identical in spirit to bug #7 (early exit from a converging loop is unsafe/meaningless once the loop body is spread across GPU threads) | Permanent, by design; commit flags the macro itself as *"to be replaced at a later time"* — i.e. considered a stopgap | +| 23 | Missing variables in `private()`/`DO_LOCALITY(local(...))` clauses causing suspected **data races** in the BBL viscosity kernel | Unmerged branch `origin/merge-omp-debug`, commits `7a51e5fb3` "cdrag locality fixes?", `fc4068582` "Private D_vel_[pm]wq" — question mark in the commit subject signals this was exploratory/unconfirmed debugging | Add `cdrag`, `cdrag_sqrt`, `cdrag_sqrt_H`, `cdrag_sqrt_H_RL`, `D_vel_p`, `D_vel_m` to `private()`/`DO_LOCALITY(local(...))`/`DO_LOCALITY(local_init(...))` clauses of the `target teams loop collapse(2) thread_limit(128)` region in `MOM_set_viscosity.F90:803-812` | **Own-code porting bug class** (incomplete privatization), not a compiler bug — but a recurring hazard: any scalar written inside a `target teams`/`do concurrent` body must be explicitly privatized or it silently races | **Unmerged / in-flight** as of this writing — not yet landed on `dev/gpu` | +| 24 | Data-upload / mapping bugs introduced by refactors, mistaken for compiler bugs | Commits `6f3a42d53` "Horvisc: Grid bugfix on GPU (Leith)" (*"The Leith params introduced new conditional loops for some of the horizontal viscosity metric arrays. This mangled some of the uploads to the GPU."*), `b404caae2` "Horvisc: biharm bugfix" (*"Forgot to copy cs%biharm_const2_xx"*), `799836a54` "dev/gfdl merge: CS%ntrunc vertvisc_limit_vel bug" (*"Accidentally added to CS%ntrunc inside the loop, rather than outside."*) | Fix the missing `map`/copy or the misplaced accumulation | **Own-code bug, not a compiler bug** | Fixed; included here specifically as a *negative example* — don't misattribute application bugs to nvfortran | + +--- + +## 1a. Symptom-signature → decision-rule index (for a debugging agent) + +Match an observed failure against a **symptom signature** in the middle column, then apply the +**decision rule**. `SILENT` = wrong answer with **no crash and no diagnostic** (the dangerous class). + +| # | Symptom signature (what you observe) | Decision rule (what to do) | +|---|---|---| +| 1 | Runtime error / illegal-address / device-dispatch fault the moment an `elemental` **type-bound** EOS proc (`this%…`) is invoked inside a `do concurrent`/`target` region | Call a free `_loc` function that takes **no `this`**; keep the type-bound proc as a thin host wrapper | +| 2 | No crash; only the `! implicit copy of `this`` comment — a `class(...)` actual still appears in a device loop | Port the remaining `this`-taking call (incl. `density_anomaly_elem_buggy_Wright`) to a `_loc` variant so `this` leaves the loop; else accept the copy | +| 3 | Build error or wrong reduction value when the `reduce`/`!$omp` target is an **array element** `itmp(i,j)` | Reduce into a **named scalar**, assign the scalar into the array element after the loop | +| 4 | Link/runtime "unsupported intrinsic" **or** wrong value from `modulo()` on the device | Replace `modulo(e,3)` with arithmetic `e_a - e_r*3` | +| 5 | `SILENT` bit-level divergence vs host in `x**(1.0/n)` (lowered to `exp((1/n)*log x)`) | Use fixed-iteration Newton `nth_root` (only `* + /`) | +| 6 | `SILENT` wrong numeric result, **only on NVHPC 25.9**, on the meridional tracer-flux face loop, when that loop is a `do concurrent` | **Do not** convert to `do concurrent`; keep `!$omp target teams loop collapse(2) private(...)` | +| 7 | `SILENT` wrong answer in the tracer insertion-sort under **NVHPC 25.11**, when the inner loop uses an early `exit` | Replace `exit` with an `if`-guarded body; **never** `exit`/`return`/`cycle` inside a device-offloaded loop | +| 8 | Kernel launches far fewer teams/blocks than the OpenACC equivalent (e.g. **17 vs ~238**); large perf loss, correct answer | Add manual `num_teams(ceiling(real(tile_iters)/128.))` | +| 9 | Wrong answers (**not** merely slow) when `ratio_max`/`flux_elem` are **not inlined** into a device region | Force inlining (`!DIR$ ATTRIBUTES FORCEINLINE`, or historically `-Minline=name:…`) | +| 10/11 | Build fragility from a hand-maintained `-Minline=name:…` list; or poor `-O2` perf of blocked mass-flux | Source-level `!DIR$ ATTRIBUTES FORCEINLINE :: name` (current) | +| 12 | *(no observable signature — undocumented `thread_limit(128)` removal)* | None; flagged for follow-up | +| 13 | Segfault in a `!$omp target`/`parallel loop` region wrapping a **k-recurrence** in `PressureForce_FV_Bouss` | Rewrite as `do concurrent (j=…, I=…)` | +| 14/15 | Assorted crashes/miscompiles in `!$omp target` regions that are equally expressible as `do concurrent` | Prefer `do concurrent` | +| 16 | **Crash** on Stellar **A100 + NVHPC 25.5** from `map(to: eta_bt) if (present(eta_bt))` (optional dummy) in `find_eta_2d` | Delete the conditional `map` of the optional arg; rely on the caller's mapping | +| 18 | ~2× GPU wall time dominated by "attaching/detaching" struct-of-arrays member arrays | Flatten arrays-of-derived-type-members into flat indexed arrays | +| 19 | Build error: compiler rejects `do concurrent (...) local(...)`/`reduce(...)` | Wrap specifiers in `DO_LOCALITY(...)`; feature-detect via `HAVE_FC_DO_CONCURRENT_LOCAL` | +| 20 | gcc/gfortran parse error on compact one-line `do concurrent (...) ; stmt ; enddo` | Expand/indent into multi-line form | +| 21 | `makedep` fails on `#define foo(x)` (empty body) or a valueless `-DMACRO` | Apply the makedep parser patch (`6474597b1`) | +| 22 | *(compile-time knob, not a failure)* GPU build wants whole-domain blocks / must skip CPU-only early-exit convergence tests | `#ifdef __NVCOMPILER_OPENMP_GPU` → block size 0 and drop `if(.not.domore) exit` | +| 23 | Suspected **data race** / nondeterministic wrong answers in the BBL viscosity `target teams loop` | Add **every** scalar written in the body (`cdrag*`, `D_vel_p/m`, …) to `private()`/`DO_LOCALITY(local[_init])` | +| 24 | Wrong answers on GPU appearing right after a refactor, resembling a compiler bug | **First** check for a missing `map`/copy or a misplaced accumulation in your own diff — *before* blaming nvfortran | + +--- + +## 2. What's a genuine nvfortran/NVHPC compiler bug vs. OpenMP semantics/perf tuning + +Sweeping the whole set above, three clusters emerge. Keep them separate — conflating them is the +single biggest risk of this catalogue being misread. + +### 2.1 Genuine compiler bugs (miscompilation/crash, version-identifiable) + +- **#1 / #2** — polymorphic `class(...) :: this` dispatch inside `do concurrent`/target regions: + runtime errors (fixed via `_loc` free functions) and a residual implicit copy (unfixed). + **Verified nuance on #2 ("`this` unused" is only partly true).** Reading the five loop bodies: + - `MOM_EOS_Wright.F90:1114,1147` and `MOM_EOS_Roquet_rho.F90:817` (the `calculate_density_derivs_*` + routines) call the `..._loc` free function inside the `do concurrent` and **never reference `this`** + — the pure "implicit copy even though unused" case. + - `MOM_EOS_Wright.F90:1008,1048` (`calculate_density_array_2d/3d_buggy_Wright`) are different: the + `if (present(rho_ref))` branch still calls `density_anomaly_elem_buggy_Wright(this, T, S, p, rho_ref)` + — i.e. `this` **is** syntactically passed into the device loop, because **no `_loc` variant of the + anomaly function exists** (only `density_elem_buggy_Wright_loc`, used in the `else` branch, drops it). + The anomaly function itself never reads `this` (it uses module-level coefficients `a0,a1,b0,c0,…`), so + the argument is *dead*, but the compiler still materialises the copy because a `class(...)` actual + appears in the call. So the fix here is not "the copy is spurious"; it is "port the anomaly path to a + `_loc` free function too, then `this` disappears from the loop." Whether that removes the copy is + unverified (no build was run). +- **#3** — `do concurrent` reduction into an array element instead of a named scalar. +- **#6** — NVHPC 25.9: `do concurrent` gives a **silently wrong numerical result** (no crash) on the + meridional tracer-flux loop in `MOM_tracer_hor_diff.F90`. This is the scariest class of bug because + it doesn't crash — it silently corrupts answers. +- **#7** — NVHPC 25.11: early `exit` from a loop nested inside `do concurrent` gives wrong answers. +- **#8** — OpenMP runtime under-launching teams (17 instead of ~238) relative to the equivalent + OpenACC code — a scheduling/heuristic bug in the runtime, not the compiler proper, but still a + genuine defect requiring a manual `num_teams` override. +- **#9** — (historical) without explicit inlining of `ratio_max`/`flux_elem`, OpenMP-compiled results + were *incorrect*, not just slow — points at a miscompilation of the un-inlined cross-procedure call + inside a device region. +- **#13** — segfault in a `!$omp target` region wrapping a recurrence loop. +- **#16** — A100 + NVHPC 25.5 crash from mapping an **optional** dummy argument + (`map(to: eta_bt) if (present(eta_bt))` in `find_eta_2d`, `MOM_interface_heights.F90`); this one is a + **confirmed crash**, fixed by removing the transfer in `2108e0eba`. +- **#17** — *(downgraded — see the corrected table row 17).* Originally presented as "the same + `if(present())+map` pattern flagged suspect a release earlier and never re-enabled." That framing does + **not** survive verification: in `f74525ae8` only one of the four commented `if(...)` blocks used + `present()`, and that one (`if(present(pbce))`) was **enabled the same day** (`d4ba8d69d`) and later + removed by a plain **refactor** (`9bfe7d358`), not because of a crash. So #17 is **not independent + evidence** that `present()+map` is a compiler bug — bug #16 remains the only *demonstrated* instance. + The construct is identical, however, which is why the foreshadowing link below still stands. + > **Resolved (2026-07-14):** *"We lose present()"* means the Fortran `present(optional_arg)` + > **intrinsic**, not the OpenACC `present()` data clause. The `f74525ae8` diff contains **no** + > OpenACC `present()` data clauses anywhere; what it does contain are commented-out conditional + > OpenMP maps of the form `!!!$omp target enter data if(present(pbce)) map(to: pbce)` — the author + > tried intrinsic-`present()` conditional maps and disabled them. Both meanings of `present()` are + > therefore the same meaning here, and the #17-↔-#16 foreshadowing link **stands**: `f74525ae8` + > disabled the very construct (`map(to: X) if(present(X))`) that later crashed an A100 under + > NVHPC 25.5 and was removed outright by `2108e0eba`. + +### 2.2 OpenMP/`do concurrent` semantics constraints (not bugs — the standard genuinely requires this) + +- **#3** (also listed above) is arguably standards-conformant: `do concurrent` reductions must name a + scalar reduction variable; MOM6's fix (reduce into a scalar, then store) is the *correct* idiom, not + a workaround for broken behavior. +- **#22**'s early-exit removal under `__NVCOMPILER_OPENMP_GPU`: an `exit`/early-termination + convergence test is meaningless once loop iterations are spread across GPU threads that don't share + loop-carried state — this is an inherent semantic mismatch between "iterate until converged" serial + algorithms and "all iterations execute independently" parallel loops, not a compiler defect. +- **#19/#20** are cross-compiler portability guards (`DO_LOCALITY`, GCC formatting), not nvfortran bugs + at all — several other compilers are the ones lacking the feature. + +### 2.3 Performance-only tuning (correct either way, chosen for speed) + +- **#8**'s `num_teams` computation (once the under-launch defect is worked around, the specific + formula `ceiling(tile_size/128.)` is a tuning choice). +- **#11**'s switch from `!NVF$ INLINE` to `!DIR$ ATTRIBUTES FORCEINLINE`, explicitly for an "-O2 perf" + win (though see the correctness caveat cross-referenced from #9). +- **#12**'s removal of `thread_limit(128)` (undocumented reason — flagged, not resolved). +- **#18**'s struct-of-arrays flattening (halved GPU time; a memory/attach-cost optimization). +- **#14/#15**'s general preference for `do concurrent` over `!$omp target` (partly reliability, partly + simplicity — commit messages don't quantify a perf delta). + +--- + +## 3. nvfortran/NVHPC versions referenced in source or commit history + +| Version | Where mentioned | What broke | +|---|---|---| +| **NVHPC 25.5** | Commit `2108e0eba` | Crash on **Stellar A100** GPUs from `map(to: eta_bt) if (present(eta_bt))` in `find_eta_2d` | +| **NVHPC 25.9** | `MOM_tracer_hor_diff.F90:1464` | `do concurrent` gives a wrong numerical result on the meridional tracer-flux face loop; kept as `!$omp target teams loop collapse(2)` instead | +| **NVHPC 25.11** | Commit `e23d6a7b1` | Early `exit` inside a loop nested in `do concurrent` produces wrong answers in the tracer insertion-sort | + +No other nvfortran point-release is named in-tree. The `__NVCOMPILER_OPENMP_GPU` predefined macro +(used at `MOM_continuity_PPM.F90:1406` et al. and `MOM_CoriolisAdv.F90:2094`) is version-agnostic — it +detects "compiling for NVIDIA GPU OpenMP offload" in general, not a specific release, and its commit +(`93dbbd36e`) explicitly calls it a stopgap: *"to be replaced at a later time."* + +The **ifort** bit-reproducibility issue in commit `5f413739b` ("Barotropic: frhat[uv] HYBRID repro +fix") is **not** an nvfortran bug — included here only as a contrast case: *"it seems possible that +Intel has added a reduction-like optimization, even at -O0"* in a loop-fission refactor. Cross-compiler +bit-repro bugs are a distinct risk category from the NVHPC-specific ones above. + +--- + +## 4. Directive / build-flag reference + +| Directive / flag | Purpose | Where used | Anchor | +|---|---|---|---| +| `!$omp declare target` | Marks a `pure`/`elemental` helper as device-callable so it can be invoked from inside a `do concurrent`/`target` region without a host round-trip | 21 occurrences across `MOM_coms.F90`, `MOM_intrinsic_functions.F90`, `MOM_set_viscosity.F90`, `MOM_vert_friction.F90` | e.g. `MOM_intrinsic_functions.F90:51` (`cuberoot`), `:133` (`nth_root`), `:181` (`rescale_cbrt`), `:246` (`descale`); `MOM_vert_friction.F90:437,2101,2611` | +| `!$omp target teams num_teams(N)` | Manually sets the team count because the OpenMP runtime under-launched (bug #8) | `MOM_continuity_PPM.F90:707,1811` | `nteams = ceiling(real((j_end-j_start+1)*(i_end-i_start+1))/128.)` computed just above each site (`:701`, `:1806`) | +| `thread_limit(N)` | Caps threads/team; used once in the current tree, was removed elsewhere (bug #12) | `MOM_set_viscosity.F90:803` (`thread_limit(128)`); removed from `MOM_continuity_PPM.F90` by `4e3f1b758` | — | +| `collapse(2)` | Flattens two loop levels into one iteration space for a `target teams loop`/`parallel loop` | Pervasive — `MOM_vert_friction.F90:737,938,1223,1255`; `MOM_set_viscosity.F90:803`; `MOM_tracer_hor_diff.F90:1465` | — | +| `!DIR$ ATTRIBUTES FORCEINLINE :: name` | Forces inlining of a cross-procedure call inside a device loop; supersedes `!NVF$ INLINE`/`-Minline=name:...` (bugs #9-#11) | `MOM_continuity_PPM.F90:1086` (`flux_elem`), `:1149` (`flux_elem_OBC`) | Introduced by `93dbbd36e` | +| `-Minline=name:ratio_max,name:flux_elem` (build flag, **not** in-tree) | Historical mandatory-inline requirement before the source-level directive existed | Commit `3cb184edd` message | Superseded — no longer needed once `!DIR$ ATTRIBUTES FORCEINLINE` is in source | +| `DO_LOCALITY(X)` macro | Conditionally applies `do concurrent` locality specifiers (`local`, `local_init`, `reduce`) only if the compiler supports them | `src/framework/do_concurrent_compat.h`; used in `MOM_continuity_PPM.F90` (11×), `MOM_CoriolisAdv.F90` (42×), `MOM_barotropic.F90` (5×), `MOM_tracer_hor_diff.F90` (9×), `MOM_tracer_advect.F90` (8×), `MOM_coms.F90` (3×), `MOM_vert_friction.F90` (6×), `MOM_set_viscosity.F90` (6×), `MOM_sum_output.F90` (4×) | Defined by `HAVE_FC_DO_CONCURRENT_LOCAL` (below) | +| `HAVE_FC_DO_CONCURRENT_LOCAL` | Autoconf-detected macro: does this Fortran compiler support `do concurrent (...) local(...)`? | `ac/m4/mom6_fc_do_concurrent_local.m4`, invoked from `ac/configure.ac:172` | Feeds `DO_LOCALITY(X)` — see `d2a72eddd`, `cd178dd52` | +| `__NVCOMPILER_OPENMP_GPU` (predefined macro, not MOM6-defined) | Detects "compiling for NVIDIA GPU via OpenMP offload" to switch default block sizes (`0` vs `32/4/1`) and to disable CPU-only early-exit convergence checks | `MOM_continuity_PPM.F90:1406,1434,1442,2416,2443,2450,3120`; `MOM_CoriolisAdv.F90:2094` | Introduced by `93dbbd36e`; explicitly called a stopgap in that commit's message | +| `omp_offload` (optional Fortran argument, not a directive) | Forwarded to `mpp_do_group_update` so halo exchanges use device-resident (GPU-aware MPI) buffers | `config_src/infra/FMS2/MOM_domain_infra.F90:1143`, passed `.true.` at ~25 call sites | See `03-openmp-mapping.md`, `11-halos-domains.md` (per `00-architecture.md` §7.3) | + +--- + +## 5. Chronology (oldest → newest, by commit) + +1. `f74525ae8` (2024-12-06) "Transition OpenACC to OpenMP" — *"We lose present() but overall it + seems to work."* Added four commented `!!!$omp target enter data if(...) &` blocks to + `MOM_PressureForce_FV.F90`; **only one used `present()`** (`if(present(pbce))`). That one was + **enabled the same day** by `d4ba8d69d` "OpenMP: PBCE on GPU" and later removed by the refactor + `9bfe7d358` (2025-01-14). "We lose present()" refers to the Fortran **`present()` intrinsic**, not + the OpenACC data clause — the diff has no OpenACC `present()` clauses, only the commented-out + `if(present(pbce))` conditional maps (see table row 17). Corrected from the original "commented + out, never enabled" claim. +2. `5274c3a8e` (2025-10-23) "fix segfault" — `omp target`+`parallel loop` around a k-recurrence + crashed; replaced with `do concurrent`. +3. `1865612de` (2025-11-27) "Convert structs of arrays to flat arrays" — attach/detach cost halved + GPU time in `MOM_tracer_hor_diff`. +4. `9aea28954` (upstream `dev-gfdl`, merged) "cuberoot: Replace modulo() with arithmetic ops" — + `modulo()` not implemented on all platforms including NVIDIA GPU. +5. `0f05b360f` (2025-08-07) "PGF: Convert omp target region to do concurrents" — *"Some compilers do + not handle the omp target syntax very well."* +6. `3cb184edd` (2026-03-17) "use openmp instead of openacc" — mandatory `-Minline` requirement first + documented. +7. `5b5f6b2b1` (2026-03-17) "add teams spec to problematic target region" — manual `num_teams` to fix + OpenMP runtime under-launching teams (17 vs. ~238). +8. `4e3f1b758` (2026-05-06) "add !NVF\$ INLINE to ratio_max flux_elem" — source-directive alternative + to `-Minline=name:...`; also drops `thread_limit(128)` from the `num_teams` region. +9. `e8b0ecfbf` (2026-05-01) "omp target teams loop -> do concurrent". +10. `2108e0eba` (2026-04-30) "Remove eta_bt transfer from find_eta_2d that was causing crashes on + stellar A100s with nvfortran 25.5". +11. `7c7af5572` / `52a1b3954` — EOS `_loc` free-function pattern for Roquet_rho/Wright polymorphic + `this` runtime errors. +12. `93dbbd36e` (Kblock continuity reconstruction) — `!NVF$ INLINE` → `!DIR$ ATTRIBUTES FORCEINLINE`; + introduces `__NVCOMPILER_OPENMP_GPU` macro for block-size defaults and early-exit removal. +13. `e23d6a7b1` (2026-03-31) "swap early exit to if guard in insert sort" — NVHPC 25.11 wrong answers + from early `exit`. +14. `MOM_tracer_hor_diff.F90:1464` NVHPC 25.9 `do concurrent` wrong-result comment (commit not + independently identified by hash in the sweep; comment is in the current merged source). +15. *(unmerged, in-flight)* `origin/merge-omp-debug`: `7a51e5fb3`/`fc4068582` — suspected missing + `private()`/`DO_LOCALITY(local(...))` variables in the BBL viscosity `target teams loop` region. + +--- + +## 6. Explicitly NOT a compiler bug (contrast cases, so they aren't miscatalogued later) + +- **`buggy_Wright_EOS` / "buggy" naming** (`MOM_EOS_Wright.F90:5,43` etc.) — this is an **upstream + MOM6 legacy EOS variant name**, predating the GPU port, deliberately retaining old science-level + arithmetic bugs (*"a poor implementation (missing parenthesis and bugs)"*) for backward answer + reproducibility. It is unrelated to nvfortran; don't confuse "buggy" in the type name with an + nvfortran defect. (It happens to be one of only two EOS forms ported to `_loc` free-function form — + see item #1 — which is why it appears throughout this catalogue.) +- **`5f413739b`** ifort bit-repro regression — a different compiler (Intel), included in §3 only as a + contrast case. +- **`6f3a42d53`, `b404caae2`, `799836a54`** (item #24) — missed `map`/copy and misplaced accumulation + bugs introduced by the GPU-porting authors themselves, not by the compiler. +- **`e5444b4e5`, `d2a72eddd`, `cd178dd52`** (`DO_LOCALITY`, GCC formatting) — these guard **against + compilers that lack a feature nvfortran has**, not against an nvfortran defect. + +--- + +## 7. Cross-references + +- `00-architecture.md` §7.1 (EOS polymorphism), §7.5 (this catalogue's origin note), §5 (k-blocking, + `__NVCOMPILER_OPENMP_GPU` block-size defaults). +- `04-do-concurrent-patterns.md` §3 (`num_teams` category A/B), §4.4 (array-element reduction bug), + §5.1-5.2 (early exit, `modulo()`, polymorphic dispatch narrative). +- `06-eos-layer.md` — full detail on the `_loc` free-function pattern and which EOS forms remain + polymorphic/unported. +- `08-cross-module-inlining.md` §3 (the exact `-Minline` requirement), §5 (nvfortran-refused device + constructs). +- `01-memory-control-structures.md` — struct-of-arrays flattening (item #18) in the CS-member context. +- `03-openmp-mapping.md` §1.4 ("Balance bugs and their fixes") for the broader map/unmap bug history + that items #16, #17, #24 are drawn from. + +--- + +## Verification notes + +Independent Opus verification against source + `git` on `dev/gpu` (no build/run performed). + +**Confirmed verbatim / hash-accurate (spot-checked all 24 quick-index rows and the §4 directive table):** + +- **Comments, verbatim:** `MOM_tracer_hor_diff.F90:1464` `! this gives wrong result when using do + concurrent on NVHPC 25.9`; `:962` `! nvfortran do concurrent cannot reduce array elements`; all five + `! NOTE: There is an implicit copy of `this` which cannot yet be prevented.` sites + (`MOM_EOS_Wright.F90:1008,1048,1114,1147`, `MOM_EOS_Roquet_rho.F90:817`); `buggy_Wright` + "poor implementation (missing parenthesis and bugs)" at `MOM_EOS_Wright.F90:5`, type at `:43`. +- **Commit messages, verbatim:** `e23d6a7b1` ("NVHPC 25.11 didn't like the early exit and would give + wrong answers"), `5b5f6b2b1` (17-vs-238 teams), `3cb184edd` (MANDATORY `-Minline`), `4e3f1b758` + (`-Minline=pragma` alternative **and** the `-…thread_limit(128)` → `+…` drop, confirmed in the diff), + `93dbbd36e` ("remove nvf inline and replace with intel forceinline … at -O2" and the + `__NVCOMPILER_OPENMP_GPU` block-size note), `2108e0eba` (Stellar A100 / nvfortran 25.5), + `5f413739b` ("Intel has added a reduction-like optimization, even at `-O0`"), and every hash in + §6/§24 (`6f3a42d53`, `b404caae2`, `799836a54`, `1865612de`, `5274c3a8e`, `0f05b360f`, `e8b0ecfbf`, + `9aea28954`, `d2a72eddd`, `cd178dd52`, `e5444b4e5`, `6474597b1`, `52a1b3954`). +- **Anchors, exact:** `FORCEINLINE` at `MOM_continuity_PPM.F90:1086,1149`; `ratio_max` `pure function` + at `:3086` with **no** inline directive; `num_teams` at `:707,1811` with `nteams` at `:701,1806`; + `thread_limit(128)` present at `MOM_set_viscosity.F90:803`; `!$omp declare target` at + `MOM_intrinsic_functions.F90:51,133,181,246` and `MOM_vert_friction.F90:437,2101,2611` (21 total + across the four named files); `modulo()` comment at `MOM_intrinsic_functions.F90:232`; all eight + `__NVCOMPILER_OPENMP_GPU` guards (`MOM_continuity_PPM.F90:1406,1434,1442,2416,2443,2450,3120`; + `MOM_CoriolisAdv.F90:2094`); `DO_LOCALITY` counts (continuity 11, CoriolisAdv 42, tracer_hor_diff 9); + `omp_offload` dummy at `MOM_domain_infra.F90:1143`. `e23d6a7b1`'s `exit`→`if`-guard rewrite and + `5274c3a8e`'s `!$omp target`→`do concurrent` rewrite both match the described workaround. +- **Branch `origin/merge-omp-debug`:** `7a51e5fb3` "cdrag locality fixes?" (question mark present) and + `fc4068582` "Private D_vel_[pm]wq", both by Marshall Ward, both touch `MOM_set_viscosity.F90`; + the added `local`/`local_init`/`private` variables (`cdrag_sqrt`, `cdrag_sqrt_H`, `cdrag_sqrt_H_RL`, + `cdrag`, `D_vel_p`, `D_vel_m`) match. Still unmerged on `dev/gpu`. + +**Corrected:** + +1. **Row 2 / §2.1** — the blanket "`this` is unused inside the device loop" is only true for + `Wright:1114,1147` and `Roquet_rho:817` (which call `_loc`). At `Wright:1008,1048` the + `present(rho_ref)` branch **passes `this`** to `density_anomaly_elem_buggy_Wright` (no `_loc` variant + exists); the argument is dead inside that function but is still syntactically present in the loop. + Fix framing added. +2. **Row 17 / §2.1 #16-#17 / §5 item 1** — the original "**four** `if(present(...)) map(to:...)` lines + left commented out, **never enabled**, foreshadowing bug #16" was substantially wrong. In + `f74525ae8` only **one** of the four commented blocks uses `present()` (`if(present(pbce))`; the + others are `if(use_EOS)`, `if(use_p_atm)`, `if(.not. use_p_atm)`). That one block was **un-commented + and enabled the same day** by `d4ba8d69d` "OpenMP: PBCE on GPU", then removed ~5 weeks later by the + **refactor** `9bfe7d358` "PF: Move pbce and eta management out of fn" — not by a crash. No such line + exists in current source. "We lose present()" does refer to the Fortran `present()` intrinsic — the + diff holds no OpenACC `present()` data clauses, only the commented-out `if(present(pbce))` + conditional maps — so the construct `f74525ae8` disabled is the same one that later crashed in bug + #16. + +**Independent completeness sweep:** grepped `src/` and `config_src/` for nvfortran/NVHPC/nvidia/ +"wrong answer|result"/"didn't like"/"cannot yet"/segfault/crash comments. No **nvfortran/GPU** bug +comment is missing from the catalogue. Two non-nvfortran comments were deliberately excluded as +out of scope: `MOM_diagnostics.F90:305` ("some compiler options can force at least one iteration…" — +a legacy ANSI-F77 loop-trip-count workaround, not GPU) and `mom_cap.F90:85` ("Model does not compile +with `use ESMF, only:`" — an ESMF module-use quirk, not nvfortran). + +**Confidence:** High for every verbatim comment, commit-message quote, hash, and file:line anchor +(all directly checked). High for the Row 2 and Row 17 corrections (checked the loop bodies and the +`f74525ae8`→`d4ba8d69d`→`9bfe7d358` chain directly). The meaning of "We lose present()" is settled: +it is the Fortran intrinsic, not the OpenACC data clause. The catalogue's "Still open?" +statuses remain as-reported (no build/run was performed to re-test them), per the document's own method +note. diff --git a/knowledge/gpu-knowledge/14-vertical-physics-ale-status.md b/knowledge/gpu-knowledge/14-vertical-physics-ale-status.md new file mode 100644 index 0000000..a62a17b --- /dev/null +++ b/knowledge/gpu-knowledge/14-vertical-physics-ale-status.md @@ -0,0 +1,714 @@ +# Vertical physics (diabatic) and ALE remap/regrid: porting status (dev/gpu) + +> Drills into architecture doc §1, §4.3, §6.3. Scope: `src/parameterizations/vertical/`, +> `src/parameterizations/lateral/MOM_mixed_layer_restrat.F90` / +> `MOM_thickness_diffuse.F90`, and `src/ALE/`. This is the **largest remaining porting +> surface** on `dev/gpu`: the entire vertical-mixing (diabatic) stack and the ALE +> vertical-Lagrangian remap/regrid machinery are essentially untouched on mainline, with +> a scatter of in-flight, non-converged branches attacking pieces of it. Read this before +> starting any work in these directories. + +--- + +## 1. Mainline status: confirmed host-only (`git diff --numstat dev-gfdl...dev/gpu`) + +| File | + | − | Status | +|---|---|---|---| +| `src/parameterizations/vertical/MOM_set_diffusivity.F90` | 0 | 0 | **untouched** | +| `src/parameterizations/vertical/MOM_CVMix_KPP.F90` | 0 | 0 | **untouched** | +| `src/parameterizations/vertical/MOM_energetic_PBL.F90` | 0 | 0 | **untouched** (mainline only) | +| `src/parameterizations/lateral/MOM_mixed_layer_restrat.F90` | 0 | 0 | **untouched** | +| `src/ALE/MOM_regridding.F90` | 0 | 0 | **untouched** | +| `src/ALE/MOM_remapping.F90` | 0 | 0 | **untouched** | +| `src/parameterizations/vertical/MOM_diabatic_driver.F90` | 3 | 0 | cosmetic only (device buffer sync around one diagnostic) | +| `src/ALE/MOM_ALE.F90` | 3 | 0 | cosmetic only (same pattern) | +| `src/parameterizations/lateral/MOM_thickness_diffuse.F90` | 3 | 0 | cosmetic only (same pattern) | +| `src/parameterizations/vertical/MOM_kappa_shear.F90` | 2 | 1 | trivial: CPU `!$OMP parallel do` `shared()` clause fix (data-race bugfix, not a GPU port) | +| `src/parameterizations/vertical/MOM_diabatic_aux.F90` | 13 | 12 | **partial**: one subroutine (`find_uv_at_h`) ported | +| `src/parameterizations/vertical/MOM_vert_friction.F90` | 775 | 184 | ported (merged, reference pattern — see architecture doc §4.3) | +| `src/parameterizations/vertical/MOM_set_viscosity.F90` | 501 | 384 | ported (merged, reference pattern) | + +The three "3-line" hits (`MOM_diabatic_driver.F90`, `MOM_ALE.F90`, `MOM_thickness_diffuse.F90`) are +all the *identical* pattern — wrapping a `call find_eta(...)` with manual +`!$omp target update to(h)` / `!$omp target enter data map(alloc: eta)` / +`!$omp target exit data map(from: eta)` so a host-side diagnostic call can read a +device-resident `h`. E.g. in `MOM_diabatic_driver.F90`: + +```fortran + if (CS%id_e_predia > 0) then + !$omp target update to(h) + !$omp target enter data map(alloc: eta) + call find_eta(h, tv, G, GV, US, eta, dZref=G%Z_ref) + !$omp target exit data map(from: eta) + call post_data(CS%id_e_predia, eta, CS%diag) + endif +``` + +This is plumbing around one specific diagnostic hook (`find_eta` itself was ported separately, +see `src/core/MOM_interface_heights.F90` and branch `find-eta-gpu`/`find-eta-merge` below) — it is +**not** evidence of any diabatic/ALE porting. `MOM_set_diffusivity.F90`, `MOM_CVMix_KPP.F90`, +`MOM_energetic_PBL.F90`, `MOM_mixed_layer_restrat.F90`, `MOM_regridding.F90`, `MOM_remapping.F90` +have **zero** diff against `dev-gfdl` — verified by empty `git diff --numstat` output (not merely +absent from a grep) for each file individually. + +The one real (if narrow) exception is `MOM_diabatic_aux.F90::find_uv_at_h` — see §2.4. + +--- + +## 2. The diabatic driver: structure, dispatch, and the heavy column kernels + +### 2.1 Top-level dispatch (`MOM_diabatic_driver.F90`) + +- `diabatic` (`:279`, `end` `:530`) — top-level entry called from `MOM.F90` (`step_MOM_thermo`). + Chooses between an ALE-coordinate path and a legacy layered path. +- `diabatic_ALE_legacy` (`:535`–`:1242`) — older ALE-mode implementation, still present. +- `diabatic_ALE` (`:1247`–`:1873`) — current ALE-mode implementation (~630 lines). +- `layered_diabatic` (`:1877`–`:2868`) — non-ALE (isopycnal/layered) implementation (~1000 lines). +- `adiabatic` (`:2905`) — no-mixing pass-through path. + +All three dispatch routines (`diabatic_ALE_legacy`, `diabatic_ALE`, `layered_diabatic`) call, in +sequence, the same set of column-physics kernels (confirmed by grep across the file — the three +`set_diffusivity` call-site pairs below live one per routine: 694/697 in `diabatic_ALE_legacy`, +1409/1412 in `diabatic_ALE`, 2113/2116 in `layered_diabatic`): + +| Kernel | Call sites (line #s) | Module | +|---|---|---| +| `set_diffusivity` | 694, 697, 1409, 1412, 2113, 2116 | `MOM_set_diffusivity.F90` | +| `KPP_compute_BLD` / `KPP_calculate` | 757/760, 763/766, 1473/1476, 1479/1482, 2172/2175, 2178/2181 | `MOM_CVMix_KPP.F90` | +| `differential_diffuse_T_S` | 826, 2253 | `MOM_diabatic_aux.F90` | +| `energetic_PBL` / `energetic_PBL_get_MLD` | 912/915, 1563/1566 | `MOM_energetic_PBL.F90` | +| `bulkmixedlayer` | 2051, 2056, 2460 | `MOM_bulk_mixed_layer.F90` | +| `tracer_vertdiff_Eulerian` / `triDiagTS_Eulerian` | 1016/1017/1019, 1664/1665 | `MOM_diabatic_aux.F90` | +| `tracer_vertdiff` / `triDiagTS` | 2419/2420/2422, 2509/2510/2512 | `MOM_diabatic_aux.F90` | +| `regularize_layers` | 2539 | `MOM_regularize_layers.F90` | + +(`calculate_kappa_shear`/`MOM_kappa_shear.F90` is invoked one level up, from +`MOM_dynamics_split_RK2.F90`/`set_viscous_ML`, feeding `Kd_shear`/`Kv_shear` into `visc`, which +`set_diffusivity` and `energetic_PBL` then consume — so it is on the same critical path even +though it isn't a direct call from `diabatic_driver.F90`.) + +### 2.2 Which kernels are inherently serial-in-k + +All of these are **column-physics kernels**: for a fixed `(i,j)` they operate on the full +`k=1..nz` water column and cannot be vectorized/parallelized over `k` because they are recurrence +relations up or down the column: + +- **`tracer_vertdiff` / `triDiagTS`** — literally solve a tridiagonal system per column (Thomas + algorithm: forward elimination down `k`, back-substitution up `k`). This is exactly the pattern + already ported in `MOM_vert_friction.F90` (`!$omp target teams loop collapse(2)` over `(i,j)` + with a serial inner `k` loop, 3 `declare target` column kernels) — the architecture doc's + "vertvisc-style teams-loop treatment" is the template these must reuse. +- **`set_diffusivity` → `find_N2`, `find_TKE_to_Kd`** — build interface buoyancy/N² profiles top-down + and then propagate a `maxEnt`/`kb`-indexed recurrence up and down the column (see §3 excerpts: + `maxEnt(i,K) = ds_dsp1(i,K)*(maxEnt(i,K-1) + htot(i))`, `htot` accumulates across `k`). +- **`energetic_PBL`** — an iterative TKE-budget integration that walks down the column mixing + layers into an evolving mixed layer (mech_TKE/conv_PErel accumulate across `k`); see §5 for the + most advanced in-flight attempt. +- **`KPP_calculate`/`KPP_compute_BLD`** — searches down the column for the boundary-layer depth + (an OBL search + shape-function evaluation), then computes a diffusivity profile — column-local + but with a top-down search recurrence. +- **`kappa_shear`** — solves an implicit/iterative shear-instability closure per column + (`Calc_kappa_shear_vertex`), similarly recurrence-based. +- **`differential_diffuse_T_S`** — solves a diffusion-like tridiagonal in `k` for T/S separately + from the main mixing. +- **`bulkmixedlayer`** — mixed-layer entrainment/detrainment logic, inherently sequential + top-down through the mixed-layer slab. + +None of these can be flattened to `do concurrent (k,j,i)` the way continuity/CoriolisAdv were; +they need the two-level pattern already proven in `MOM_vert_friction.F90`: parallelize over +`(i,j)` (whole-column-per-thread), keep `k` as a private serial loop inside a `declare target` +column kernel. + +### 2.3 `layered_diabatic` vs `diabatic_ALE` + +Both drivers call the *same* physics kernels but differ in bookkeeping: `layered_diabatic` mixes +directly into an isopycnal grid and calls `regularize_layers` at the end to repair degenerate +layers; `diabatic_ALE` defers grid regeneration to the subsequent ALE remap step +(`ALE_regridding_and_remapping`, `MOM.F90:916`/`:1037`) and has no `regularize_layers` call. This +means **porting the column kernels once (in `MOM_set_diffusivity.F90`/`MOM_CVMix_KPP.F90`/ +`MOM_energetic_PBL.F90`) benefits all three dispatch paths** (`diabatic_ALE_legacy` included) — there +is no need to port them per-driver. + +### 2.4 The one real precedent: `find_uv_at_h` in `MOM_diabatic_aux.F90` + +This is the only genuine (if narrow) GPU port merged anywhere in the diabatic stack. Before: +plain nested `do j / do i` with `!$OMP parallel do`. After: + +```fortran + !$omp target enter data map(alloc: a_w,a_e,a_s,a_n,b1,d1,c1) + !$omp target teams loop private(sum_area,Idenom,a_w,a_e,a_s,a_n,b_denom_1,b1,d1,c1) & + !$omp map(to: ea, eb, h) map(from: u_h, v_h) + do j=js,je + do concurrent (i=is:ie) + ... + enddo + if (mix_vertically) then + do concurrent (i=is:ie) + ... + enddo + do k=2,nz ; do concurrent (i=is:ie) ! forward elimination — k stays a serial do-loop + c1(i,k) = eb(i,j,k-1) * b1(i) + ... + enddo ; enddo + do k=nz-1,1,-1 ; do concurrent (i=is:ie) ! back substitution — k stays a serial do-loop + u_h(i,j,k) = u_h(i,j,k) + c1(i,k+1)*u_h(i,j,k+1) + ... + enddo ; enddo + ... + enddo + !$omp target exit data map(release: a_w,a_e,a_s,a_n,b1,d1,c1) +``` + +This is exactly the target pattern for the rest of the diabatic stack: `!$omp target teams loop` +over `j` (or `(i,j)` collapsed), `do concurrent (i=...)` for the parallel dimension, and explicit +serial `do k=...` for the tridiagonal recurrence — nothing else in `MOM_diabatic_aux.F90` or the +rest of the diabatic stack has received this treatment yet. + +--- + +## 3. In-flight: `remotes/edoyango/port-set_diffusivity` (and siblings `port/set_diffusivity`, +`set_diffusivity-kjiarrs`) + +`git log --oneline dev/gpu..remotes/edoyango/port-set_diffusivity`: + +``` +fc8ca6c7d more kji in find_tke_to_kd +ba65ee4bf kji arrays set_density_ratios +57b119b1f ijk arrays in find_n2 +efe136545 set_diffusivity: move remaining OMP j-loop blocks out into own j-loops +fdac8e44d set_diffusivity: block j loops in add_drag_diffusivity +5d5a1d3e7 set_diffusivity: move ML_radiation, tidal_mixing, int_tides out of OMP loop +e7d7fb983 set_diffusivity: consolidate thickness_to_dz into single dz array +7978a3236 set_diffusivity: block j loops in find_TKE_to_Kd and set_density_ratios +3670d6c68 set_diffusivity: block j loops in calculate_bkgnd_mixing and find_TKE_to_Kd +02974f28d set_diffusivity: block j loops in find_N2 and find_rho_bottom +4a4290548 add j dimension to find_N2 related arrs +``` + +`git diff --numstat dev/gpu...remotes/edoyango/port-set_diffusivity`: + +``` +204 2 src/core/MOM_interface_heights.F90 +130 110 src/parameterizations/vertical/MOM_bkgnd_mixing.F90 +693 584 src/parameterizations/vertical/MOM_set_diffusivity.F90 +``` + +The superset branch `remotes/edoyango/port/set_diffusivity` (10 commits ahead of this one, adds +`add_LOTW_BBL_diffusivity` tile-index support and explicit `! TODO` markers for porting/blocking) +touches an even wider footprint — it also pulls in `MOM_isopycnal_slopes.F90` (+400/−359), +`MOM_lateral_mixing_coeffs.F90`, `MOM_MEKE.F90`, `MOM_stoch_eos.F90` — showing that +`set_diffusivity` cannot be ported in isolation; its buoyancy/N² inputs are shared with the +lateral-mixing-coefficient and MEKE machinery. + +### 3.1 What transformation is actually applied + +**This is not the blessed k-blocking template.** There is **zero** `omp target`, `do concurrent`, +or `DO_LOCALITY` anywhere in this branch's diff (`grep -c` on the diff returns 0). It is a +*preparatory* refactor only, in two parts: + +1. **j-blocking (promote scalar-`j` column routines to `nj`-wide row-blocks).** Every helper + (`find_N2`, `find_TKE_to_Kd`, `add_drag_diffusivity`, `set_density_ratios`, ...) is rewritten + from `real, dimension(SZI_(G),SZK_(GV))` (a single j-row, called once per `j` inside an + `!$OMP parallel do` over `j`) to `real, dimension(SZI_(G),SZK_(GV),nj)` (a block of `nj` rows, + dummy args promoted to `jstart`/`jend`/`nj`), and the call site is hoisted out of the per-`j` + OMP loop so it processes a whole row-block at once. Excerpt (`fdac8e44d`, `add_drag_diffusivity`): + ```fortran + -subroutine add_drag_diffusivity(h, u, v, tv, fluxes, visc, j, TKE_to_Kd, maxTKE, & + +subroutine add_drag_diffusivity(h, u, v, tv, fluxes, visc, jstart, jend, nj, TKE_to_Kd, maxTKE, & + kb, rho_bot, G, GV, US, CS, Kd_lay, Kd_int, Kd_BBL) + - integer, intent(in) :: j !< j-index of row to work on + - real, dimension(SZI_(G),SZK_(GV)), intent(in) :: TKE_to_Kd + + integer, intent(in) :: jstart, jend, nj + + real, dimension(SZI_(G),SZK_(GV),nj), intent(in) :: TKE_to_Kd + ``` + This is the row-block analogue of the `niblock/njblock/nkblock` idea from §5 of the + architecture doc, but applied by hand at the *array-dimension* level rather than through the + CS-parameter block-size machinery used in `MOM_continuity_PPM.F90`. + +2. **Array dimension reordering, `(i,k,j)` → `(i,j,k)`.** Once row-blocked, local work arrays like + `dRho_int`, `rho_0`, `dsp1_ds`, `maxEnt` are declared `(SZI_(G),nj,SZK_(GV))` instead of + `(SZI_(G),SZK_(GV),nj)`, i.e. `k` is pushed to the *last* (slowest-varying) dimension and `j` + becomes the middle dimension. Excerpt (`fc8ca6c7d`, `find_TKE_to_Kd`): + ```fortran + - real, dimension(SZI_(G),SZK_(GV),nj) :: & + + real, dimension(SZI_(G),nj,SZK_(GV)) :: & + ds_dsp1, dsp1_ds, maxEnt, rho_0, ... + ... + - do j=jstart,jend ; jj = j - jstart + 1 ; do k=2,nz-1 ; do i=is,ie + - dsp1_ds(i,k,jj) = 1.0 / ds_dsp1(i,k,jj) + + do k=2,nz-1 ; do j=jstart,jend ; jj = j - jstart + 1 ; do i=is,ie + + dsp1_ds(i,jj,k) = 1.0 / ds_dsp1(i,jj,k) + enddo ; enddo ; enddo + ``` + and loop nests are correspondingly re-ordered so `k` is the **outer** loop and `(j,i)` the + inner ones — i.e. the branch is deliberately exposing `(i,j)` as the parallel dimension pair + and isolating the serial recurrence (`maxEnt`/`kb`-indexed) to the outer `k` loop, in + preparation for eventually replacing the inner `(j,i)` nest with `do concurrent` and keeping + `k` a private serial loop — the same shape as the `find_uv_at_h` precedent in §2.4, just not + yet wired up with any device directives. + +**Verdict:** genuine, disciplined groundwork toward the vertvisc-style pattern, but pre-device — +no `omp target`/`do concurrent`/`DO_LOCALITY` has landed yet on any of the three sibling branches. + +--- + +## 4. In-flight: `remotes/edoyango/port/thickness_diffuse` + +`git diff --numstat dev/gpu...remotes/edoyango/port/thickness_diffuse`: + +``` + 1 1 pkg/CVMix-src + 1 1 src/ALE/MOM_ALE.F90 + 3 3 src/ALE/PLM_functions.F90 + 69 31 src/core/MOM_PressureForce_FV.F90 + 30 23 src/core/MOM_PressureForce_Montgomery.F90 + 62 456 src/core/MOM_density_integrals.F90 +589 0 src/core/MOM_density_integrals_s.F90 (new submodule file) +400 359 src/core/MOM_isopycnal_slopes.F90 + 4 0 src/core/MOM_stoch_eos.F90 +241 0 src/equation_of_state/MOM_EOS.F90 +158 16 src/equation_of_state/MOM_EOS_Roquet_rho.F90 + 77 0 src/equation_of_state/MOM_EOS_Wright.F90 + 92 0 src/equation_of_state/MOM_EOS_base_type.F90 + 12 1 src/parameterizations/lateral/MOM_MEKE.F90 + 47 3 src/parameterizations/lateral/MOM_lateral_mixing_coeffs.F90 +338 347 src/parameterizations/lateral/MOM_thickness_diffuse.F90 + 4 0 src/parameterizations/vertical/MOM_internal_tide_input.F90 + 4 0 src/parameterizations/vertical/MOM_set_diffusivity.F90 +``` + +This is **much further along** than `set_diffusivity` — it is the one branch in this whole survey +that has actual device directives in the vertical/lateral-mixing space (via `git log --oneline`: +`add DO_LOCALITY macro`, `port remaining loops in thickness_diffuse`, `port thickness diffuse +full`, `submodulify`/`desubmodulify` — module split into `MOM_thickness_diffuse_s.F90` submodule). +Excerpt from `add DO_LOCALITY macro` (`6d8b47efc`), showing real `do concurrent` + `DO_LOCALITY` +already in place, being tidied to use the macro instead of hand-written `local()`/`local_init()` +clauses: + +```fortran +- do concurrent (j=jstart:jend, i=is-1:ie) & +- local_init(drdiA, drdiB, drdkL, drdkR) & +- local(drdz, hg2L, hg2R, haL, haR, dzaL, dzaR, wtL, wtR, ...) ++ do concurrent (j=jstart:jend, i=is-1:ie) DO_LOCALITY(local_init(drdiA, drdiB, drdkL, drdkR)) +``` + +**EOS/mixing-coefficient dependencies pulled in:** `thickness_diffuse_full` (isopycnal/GM +height-diffusion) needs horizontal density gradients and the Fukumori-type internal wave speed +`cg1`, so this branch is forced to also port: +- `MOM_density_integrals.F90`/new `MOM_density_integrals_s.F90` — the `int_density_dz_generic_plm` + PLM pressure/density integral used for slope estimates (heavy k-blocking work visible in the log: + `int_density_dz_generic_plm: tile ... in intx_dpa/inty_dpa update`, "use 2d calc dens", "move k + loops inside"). +- `MOM_isopycnal_slopes.F90` (`calc_isoneutral_slopes`) — feeds `Slope`/`N2` to the streamfunction + limiter. +- `MOM_EOS.F90` / `MOM_EOS_base_type.F90` / `MOM_EOS_Wright.F90` / `MOM_EOS_Roquet_rho.F90` — the + same EOS `_loc`-function device pattern documented in `06-eos-layer.md`, extended/duplicated + here rather than reused, since the merged EOS port only covers Wright/Roquet. +- `MOM_MEKE.F90` / `MOM_lateral_mixing_coeffs.F90` — `cg1`/`Rd`/eddy-length inputs are shared + state consumed by `thickness_diffuse`. +- `MOM_PressureForce_FV.F90` / `MOM_PressureForce_Montgomery.F90` — touched for the same PLM + density-integral change, since both pressure-force and thickness-diffusion reuse + `int_density_dz_generic_plm`. + +So thickness_diffuse is not an isolated kernel — porting it drags in essentially the whole +density/pressure-integral subsystem. This branch shows the *cost* of that in practice: 17 files +touched to port one lateral-mixing routine. + +--- + +## 5. EPBL (`MOM_energetic_PBL.F90`): untouched on mainline, but a striking naive-port +experiment exists off-tree (`remotes/origin/epbl-3d` and siblings) + +This is a bigger and more informative in-flight effort than the branches enumerated in the +original task brief, and directly relevant to the "heavy column kernels" question in §2.2, so it +is documented here in full. + +`git log --oneline dev/gpu..remotes/origin/epbl-3d`: +``` +c954189f9 ePBL: do concurrent +c627c0d09 ePBL: Overly aggressive inlining +05c74b56b ePBL: Replace auto array size (nk=75) +99fd0e4f3 ePBL: 3D test: remove 2d/1d copies and directives +8dd5fe2d8 Test 3d version of epbl_column +5166b07aa ePBL: Submodule for MOM_wave_interface +bf394e40d ePBL debug: minor cleanups +a433f1eb4 ePBL: Remove redundant module loads in submodule +``` +`git diff --numstat dev/gpu...remotes/origin/epbl-3d`: +``` + 4 3 src/core/MOM_interface_heights.F90 + 415 3674 src/parameterizations/vertical/MOM_energetic_PBL.F90 +5031 0 src/parameterizations/vertical/MOM_energetic_PBL_smod.F90 (new submodule) + 19 0 src/parameterizations/vertical/smod.mk (submodule makefile) + 46 432 src/user/MOM_wave_interface.F90 + 475 0 src/user/MOM_wave_interface_smod.F90 (new submodule) +``` +Note the branch also submodulifies `MOM_wave_interface` (commit `5166b07aa`) — because +`get_Langmuir_Number` lives there and must be reachable/inlinable from the device kernel — and touches +`MOM_interface_heights.F90`; the port is not confined to the two EPBL files. (`git log --oneline +dev/gpu..remotes/origin/epbl-3d` is 13 commits; the 8 shown above are the most recent, ending the +lineage at `c954189f9`.) Sibling branches `epbl-debug`, `epbl-debug-3d`, `epbl-debug-submod`, +`epbl-debug-submod-github` are earlier checkpoints of the same lineage, converging on `epbl-3d`. + +**The approach taken here is deliberately the opposite of the k-blocking template**: instead of +tiling and hoisting communication, the entire per-column TKE-budget subroutine +(`ePBL_column`/`ePBL_column_3d`, thousands of lines) is marked `pure`, and the outer `(i,j)` loop +is wrapped directly in `do concurrent`, calling the whole column kernel as one opaque body. From +the final commit (`c954189f9`, message: *"The magic of do concurrent has sped this up 100x to +~2ms/step"*): + +```fortran +- !$omp target loop private(SpV_dt) +- do j=js,je ++ do concurrent (j=js:je, i=is:ie) ++ if (G%mask2dT(i,j) > 0.) then + ... +``` + +```fortran +-subroutine ePBL_column_3d(h, dz, u, v, ..., G, i, j, TKE_gen_stoch, TKE_diss_stoch, tmpval) ++pure subroutine ePBL_column_3d(h, dz, u, v, ..., G, i, j, TKE_gen_stoch, TKE_diss_stoch, tmpval) +``` + +Two nvfortran-workaround commits accompany this ("naive whole-kernel `do concurrent`" is not free): + +- **`05c74b56b` "Replace auto array size (nk=75)"** — local work arrays inside the column kernel + were declared `real, dimension(SZK_(GV)+1) :: ...` (a size derived from a derived-type member, + i.e. an automatic array with a runtime-known but non-dummy-argument extent). This was replaced + with a **hardcoded literal `75`** to get past a device-compilation limitation on automatic arrays + sized from non-argument expressions inside a `do concurrent`/`pure` procedure — a real + nvfortran-workaround worth cataloguing (see `13-compiler-workarounds.md`), and a portability + landmine (breaks silently if `GV%ke /= 75`). +- **`c627c0d09` "Overly aggressive inlining"** (commit message: *"This is too hideous to describe + in words. But I am trying to inline as much as possible."*) — manually inlines calls like + `get_Langmuir_Number`/`find_mstar` because cross-module calls from inside the `pure`/device + region don't resolve cleanly, echoing architecture-doc principle 4 ("cross-module calls inside + device loops are painful... must be inlined"). + +**Assessment:** this is real evidence that (a) EPBL *can* be parallelized over `(i,j)` as a single +opaque `pure` column kernel without restructuring its internal `k` recurrence at all (the "naive +port" contrast case referenced in the architecture doc for `bodner-naive-port`), and (b) doing so +still requires nontrivial workarounds (automatic-array sizing, manual inlining) and is not yet a +CPU-preserving k-blocked port — it is a single-target (GPU-only, `do concurrent` whole-kernel) +experiment, not obviously mergeable as-is (mainline principle 1: "preserve CPU performance"). +It has **not** touched `dev/gpu` and `MOM_energetic_PBL.F90` remains at 0/0 there. + +--- + +## 6. `MOM_set_viscosity.F90` / `MOM_vert_friction.F90` tiling branches (adjacent, in-flight) + +Three further edoyango branches continue tiling work on the *already-merged* viscosity/friction +modules (not diabatic/ALE, but immediately adjacent and using the same idiom, useful precedent): + +- `remotes/edoyango/port-set_viscous_BBL-tile` — tiles temporaries in `set_viscous_BBL` + (`TILE_SIZE_X`×`TILE_SIZE_Y`, tile-local indices), `MOM_set_viscosity.F90` +238/−176. +- `remotes/edoyango/port-set_viscous_ML-tile` — tiles `set_viscous_ML` (block-size inputs, 2D + promotion of 1D temporaries, global array transfers), +678/−476. +- `remotes/edoyango/tile-vertvisc-coef` — promotes `vertvisc_coef`/`vertvisc`/`vertvisc_remnant` + locals to 3D arrays and adds `bind(parallel,teams)` to a `do concurrent`, `MOM_vert_friction.F90` + +457/−351 (one commit literally named `claude tile vertvisc_coef` — LLM-assisted). + +These confirm the "j/i-tiling of column-local temporaries" idiom is the actively-used +transformation across all the vertical-physics branches right now (as opposed to the +niblock/njblock/nkblock CS-parameter block machinery used in `MOM_continuity_PPM.F90`) — expect +this to be the template that eventually lands for `set_diffusivity`/KPP/EPBL too. + +--- + +## 7. ALE remap/regrid: why per-column reconstruction is architecturally hard to offload + +### 7.1 Call structure + +`ALE_regridding_and_remapping` (`MOM.F90:1900`–`:2073`) is called from `step_MOM_thermo` +(`MOM.F90:916` and `:1037`, once per branch of an ALE/non-ALE conditional). Internally +(`MOM_ALE.F90`): + +- `regridding_main(CS%remapCS, CS%regridCS, ...)` generates the new target grid — dispatches via + a plain integer `select case (CS%regridding_scheme)` (`MOM_regridding.F90:1281`) over + `build_grid_HyCOM1`/`build_grid_adaptive`/z-star/sigma/rho builders. This part is *not* + polymorphic and would k-block reasonably (it's the same shape of problem as other dispatch code + already ported elsewhere). +- The remap step is a **plain host loop calling a per-column subroutine**, not any kind of + `do concurrent`: + ```fortran + do j = G%jsc-1,G%jec+1 ; do i = G%isc-1,G%iec+1 + call remapping_core_h(CS%remapCS, nz, h_orig(i,j,:), tv%S(i,j,:), nz, h(i,j,:), & + tv_local%S(i,j,:)) + call remapping_core_h(CS%remapCS, nz, h_orig(i,j,:), tv%T(i,j,:), nz, h(i,j,:), & + tv_local%T(i,j,:)) + enddo ; enddo + ``` + (`MOM_ALE.F90:745-748`; the identical pattern recurs at `:836/838` for tracers, `:1192`/`:1267` + for velocity remap, and inside `coord_rho.F90:277/279`, `coord_hycom.F90:240/241` for + coordinate-generator-internal remaps.) + +### 7.2 Why `remapping_core_h`/`Recon1d_*` resist the k-blocking template + +1. **Ragged/variable column length.** `remapping_core_h(CS, n0, h0, u0, n1, h1, u1, ...)` takes + `n0`/`n1` as *arguments*, not fixed to `GV%ke` — columns can have a different active layer count + (thin/vanished layers), and the internal sub-cell intersection (`intersect_src_tgt_grids`) + builds per-column index arrays (`isrc_start`, `isrc_end`, `isub_src`, ...) whose *sizes and + control flow depend on the column's own data*. This is fundamentally a per-column + variable-length merge/list-intersection algorithm — the opposite of the uniform, statically- + shaped stencil work `do concurrent`/SIMD lanes want. + +2. **Polymorphic dispatch (`class(Recon1d)`), the same v-table problem already catalogued for + EOS.** `Recon1d` (`src/ALE/Recon1d_type.F90:16`) is `abstract` with **eight `deferred` + type-bound procedures** (`init`, `reconstruct`, `average`, `f`, `dfdx`, `check_reconstruction`, + `unit_tests`, `destroy`, plus `init_parent`/`reconstruct_parent`), and `remapping_CS` carries + `class(Recon1d), pointer :: reconstruction` (`MOM_remapping.F90:83`). When + `CS%remapping_scheme == REMAPPING_VIA_CLASS`, the core routine dispatches through this pointer: + ```fortran + if (CS%remapping_scheme == REMAPPING_VIA_CLASS) then + call CS%reconstruction%reconstruct(h0, u0) + call CS%reconstruction%remap_to_sub_grid(h0, u0, n1, h_sub, ...) + else ! Uses the OM4-era integer-keyed select-case reconstruction functions instead + ``` + (`MOM_remapping.F90:273-300`). This is architecturally identical to the pre-port + `EOS_type`/`class(EOS_base)` situation documented in `06-eos-layer.md` §7.1: nvfortran cannot + resolve v-table dispatch on device. **The escape hatch is the OM4-era `else` path**: it is + already non-polymorphic (integer-keyed `select case`, no `class(Recon1d)` deref), so it needs no + `_loc`-style rewrite at all — a device port can simply set `remapping_scheme /= REMAPPING_VIA_CLASS` + and target the `remap_src_to_sub_grid_om4`/`remap_sub_to_tgt_grid_om4`/`build_reconstructions_1d` + functions directly. **This is exactly what `remotes/origin/jorge/diagnostics_port` does** (see §7.3): + it marks that whole OM4 call chain `!$omp declare target` and fixes the ragged sizing with a + `NK_GPU_MAX` parameter — so the earlier draft's "no branch is attempting to port `Recon1d_*`" is + **incorrect**; a preparatory device-enablement attempt exists, it just avoids `Recon1d` rather + than de-polymorphizing it. + Note: even in the *non*-polymorphic (`else`) branch, dispatch is still per-scheme via + `select case` over ~9 reconstruction kinds (PCM/PLM/PLM_hybgen/PPM_CW/PPM_H4/PPM_IH4/ + PPM_hybgen/PQM/...) inside a routine called once per column — so columns using different + schemes (rare, but the scheme is a runtime CS setting, uniform per-run in practice) would + otherwise diverge; in practice this is not the blocking issue, the deferred-procedure dispatch + and ragged sizing are. + +3. **Deep, non-inlined call chains inside the per-column kernel.** `remapping_core_h` alone calls + `intersect_src_tgt_grids`, `build_reconstructions_1d`, `remap_src_to_sub_grid[_om4]` (or the + polymorphic `CS%reconstruction%remap_to_sub_grid`), and `remap_sub_to_tgt_grid[_om4]` — roughly a + half-dozen cross-procedure calls per column (`adjust_h_sub` is present in source but currently + commented out at `:280`/`:809`), each of which would need `!$omp declare target` + guaranteed + inlining (architecture-doc principle 4) to avoid an interpreted/indirect call inside a device + loop. This is precisely the set `jorge/diagnostics_port` blanket-tags `!$omp declare target`. + +4. **Optional arguments and automatic-size locals.** `remapping_core_h` has optional dummy args + (`net_err`, `PCM_cell`) and internal automatic arrays sized by the dummy `n0`/`CS%degree+1` + (`ppoly_r_coefs(n0,CS%degree+1)`) — both patterns are awkward inside `do concurrent`/`pure` + procedures on current nvfortran (the EPBL "hardcode nk=75" workaround in §5 is exactly this + class of problem, just for a fixed-size case; ALE's is worse because the size is *not* even + fixed at `GV%ke`, it's a runtime sub-column count). + +### 7.3 Branches touching ALE files — one preparatory remap attempt, the rest incidental + +Full survey of every branch/remote with a nonzero diff against `MOM_remapping.F90` / +`MOM_regridding.F90` / `MOM_ALE.F90` / `Recon1d_type.F90`: + +| Branch | Files touched | Nature | +|---|---|---| +| most branches (`bodner-naive-port`, `port/pressureforce-benchmark_ALE`, `find-eta-gpu`, `find-eta-merge`, `find-eta-with-CS`, `feat/new-diag-manager`, `port/thickness_diffuse`) | `MOM_ALE.F90` only, 1–3 lines | The `find_eta` device-buffer-sync pattern from §1, or a one-line diag-manager hook — **not** a remap port | +| `remotes/origin/cmake/amd-flang` | `MOM_remapping.F90` 36/34 | AMD/flang build-portability fixes, not NVIDIA/nvfortran GPU work | +| **`remotes/origin/jorge/diagnostics_port`** | **`MOM_remapping.F90` 82/62** | **The one real remap-offload attempt.** Adds `!$omp declare target` to the entire OM4 per-column call chain (`remapping_core_h`, `build_reconstructions_1d`, `intersect_src_tgt_grids`, `remap_src_to_sub_grid[_om4]`, `remap_sub_to_tgt_grid_om4`, `interpolate_column`, `average_value_ppoly` — all tagged `! GPU PORT DIAGNOSTICS`) and introduces `integer, parameter, public :: NK_GPU_MAX = 500` (`:47`), rewriting the ragged automatic locals `frac_pos`/`k_src` to `NK_GPU_MAX+1` fixed size (`:1275-1277`) — the ALE analogue of the EPBL `nk=75` workaround (§5). It is a **single messy WIP commit** (`c5810df0b` "diagnostics kinda clean"); it does **not** yet wrap the `MOM_ALE.F90:745` host loop in a device region (no `omp target`/`do concurrent` added there), so the routines are device-*callable* but not yet device-*driven*. Still: this is genuine preparatory device-enablement, and it validates the "target the OM4 path, sidestep `Recon1d`" strategy (§7.2). | +| `remotes/JorgeG94/jorge/use_dp_as_real` | `MOM_ALE.F90` 243/241, `MOM_regridding.F90` 386/384, `MOM_remapping.F90` 521/519, `Recon1d_type.F90` 52/50 | Precision refactor: replace `-r8` compiler flag with explicit `real(real64)` kinds everywhere (`801689d87`, "use real64 everywhere instead of -r8") — a plausible *prerequisite* for device work (explicit kinds matter for device code per architecture-doc conventions) but **not itself an offload** | +| `remotes/edoyango/submod-conversion` | `MOM_ALE.F90` 39/1244, `MOM_regridding.F90` 45/2372, `MOM_remapping.F90` 30/2445, `Recon1d_type.F90` 6/187 | Mechanical, codebase-wide module→module+submodule split via a script (`convert_to_submodules.py`, ~290 new `_s.F90` files) for **incremental compile-speed**, not GPU-related; same submodule idiom later reused by hand in the `thickness_diffuse`/`epbl-3d` branches | +| ~all branches | `Recon1d_type.F90` 4/2 | Ubiquitous license-header swap (SPDX identifier), **not** code — ignore | + +**Conclusion for Q5 (corrected): exactly one branch — `remotes/origin/jorge/diagnostics_port` — +is attempting to make the `remapping_core_h`/OM4 reconstruction chain device-executable** +(`declare target` + `NK_GPU_MAX` fixed sizing), but it stops at making the routines *callable* on +device; **no branch yet drives them from a device loop**, and none touches the polymorphic +`Recon1d_*` path. So remapping remains the least mature corner of the effort, but it is **not** true +that "zero work, preparatory or otherwise" exists — the de-polymorphization question is effectively +answered (use the OM4 path) and the ragged-sizing question has a candidate answer (`NK_GPU_MAX`); +what's missing is the driving device loop + bitwise validation. + +> **Open (reviewed 2026-07-14):** Is the `diagnostics_port` strategy (OM4 select-case path + blanket +> `declare target` + `NK_GPU_MAX=500` fixed sizing) the right long-term direction, or a dead end? The +> review sharpened this: at `GV%ke≈75`, 500-deep per-thread private column arrays over-allocate +> device local memory ~6.7×, and several such arrays per thread will spill and crush occupancy — +> prefer sizing from the dummy argument (`size(h,3)`) or a blocked redesign. But the constraint +> forcing the fixed size is real (nvfortran rejects non-dummy-sized automatics in device `pure` +> procedures, `05c74b56b`), so the pragmatic middle is a `parameter` sized to a realistic maximum +> (e.g. 128) plus an init-time `FATAL` guard. Needs an occupancy measurement. See KNOWLEDGE.md §9. + +--- + +## 8. Prioritized remaining-work assessment + +Ranked by how directly the subsystem sits on the per-timestep critical path (called every +dynamic/thermo step, unconditionally, vs. only under specific runtime configs) and by current +in-flight momentum: + +### Tier 1 — on the critical path every thermodynamic step, must eventually port, work started + +Recommended dependency ordering: **EOS `_loc` coverage (see below) → `find_N2`/density inputs → +`set_diffusivity` → KPP/EPBL → `kappa_shear`**. The tridiagonal solves (#1) are independent and can +go first as a warm-up. + +1. **`MOM_diabatic_aux.F90` tridiagonal solves** (`tracer_vertdiff`/`triDiagTS`, `triDiagTS_Eulerian`). + - *Approach:* copy the in-file `find_uv_at_h` template verbatim (§2.4): `!$omp target teams loop` + over `j`, `do concurrent (i=is:ie)` for the parallel dimension, explicit serial `do k` for the + forward-elimination / back-substitution recurrence, with the tridiagonal work arrays + (`b1`,`c1`,`d1`) `map(alloc:)`/`map(release:)` around the region. Equivalently the merged + `MOM_vert_friction.F90` teams-loop-collapse(2) form (`:737`,`:938`,`:1223`,`:1255`, 3 + `!$omp declare target` column kernels at `:437`,`:2101`,`:2611`). + - *Hazards:* these routines take `tv%T`/`tv%S` **pointer** members of `thermo_var_ptrs` (arch-doc + §2.3) — map the target arrays, not the container; guard any diagnostic post behind `id_* > 0` + device-update (arch-doc §7.4). No restart registration on these temporaries. + - *Why first:* lowest remaining effort — the precedent already lives in the same file. +2. **`MOM_set_diffusivity.F90`** — called unconditionally from all three diabatic drivers (§2.1), + several call sites per step. + - *Approach:* finish the existing `port-set_diffusivity` lineage — its j-blocking + + `(i,j,k)`-reorder groundwork already exposes `(i,j)` as the parallel pair with `k` isolated as + the serial recurrence (`maxEnt`/`kb`), i.e. the `find_uv_at_h`/vertvisc shape minus the device + directives. Add `DO_LOCALITY`-annotated `do concurrent (j,i)` + serial-`k`, or teams-loop. + - *Dependency (must land first):* its buoyancy/N² inputs (`find_N2`, `calc_isoneutral_slopes`, + density integrals) are shared with `lateral_mixing_coeffs`/MEKE and call EOS — so the EOS `_loc` + chain and `MOM_isopycnal_slopes.F90` need device coverage first (this is exactly why the + `port/set_diffusivity` superset drags those in; §3). Prefer merging the EOS/density-integral + work from `port/thickness_diffuse` (§4) before wiring device directives here. + - *Hazards:* `set_diffusivity` writes into `visc%Kd_*` which include restart-target **pointer** + fields of `vertvisc_type` (arch-doc §2.3, §7.4) — keep those device-resident and only + `target update from` on diag/restart boundaries. +3. **`MOM_energetic_PBL.F90`** — the default boundary-layer scheme in most configs, every thermo step. + - *Approach:* the `epbl-3d` branch proves `(i,j)`-parallel `do concurrent` over a `pure` + whole-column kernel works, but as a **GPU-only, non-CPU-preserving** variant (§5). Two viable + paths: (a) accept the naive `pure`-column `do concurrent` if the "preserve CPU performance" + principle can be waived here (pending the `bodner-naive-port` precedent discussion), or (b) redo + in blessed teams-loop style. Either way reuse `epbl-3d`'s two hard-won workarounds: manual + inlining of `get_Langmuir_Number`/`find_mstar` (cross-module calls, arch-doc principle 4) and a + fixed-size replacement for the `SZK_(GV)+1` automatic column arrays. + - *Hazard:* the hardcoded `nk=75` in `epbl-3d` is a portability landmine — parameterize to `GV%ke` + or a validated `NK_MAX` before reuse. Depends on `MOM_wave_interface` being device-reachable + (submodulified on `epbl-3d`). +4. **`MOM_CVMix_KPP.F90`** — alternative/complementary boundary-layer scheme, same call frequency as + EPBL when enabled. **Zero in-flight work found** — the least-started "always runs" kernel. + - *Approach:* same two-level column pattern (OBL-depth search + shape function is a top-down + column recurrence, §2.2). Hazard: KPP calls into the external CVMix package (`pkg/CVMix-src`); + those cross-library calls must be `declare target` + inlinable or duplicated, a bigger inlining + surface than the other Tier-1 items. +5. **`MOM_kappa_shear.F90`** — feeds `Kd_shear`/`Kv_shear` consumed by `set_diffusivity`/EPBL; only + change on `dev/gpu` is an unrelated CPU OpenMP data-race fix (`shared()` clause). No in-flight GPU + work. *Approach:* per-column iterative closure (`Calc_kappa_shear_vertex`) → same teams-loop shape; + port after `set_diffusivity` since its output is that routine's input. Note it is invoked from the + **dynamics** side (`set_viscous_ML`), so its device data must be live across the dycore→diabatic + boundary. + +### Tier 2 — on the critical path but conditional on runtime config, or ALE-specific +6. **ALE remap (`MOM_remapping.F90`/`Recon1d_*`)** — runs every timestep when ALE mode is active + (the default coordinate mode in most modern MOM6 configs), so functionally Tier-1 in practice, + but ranked here because it is **architecturally the hardest problem in this whole survey** + (ragged per-column sizing + polymorphic `Recon1d` dispatch + deep call chains, §7). **Not zero + in-flight work** (corrected): `remotes/origin/jorge/diagnostics_port` has already tagged the OM4 + per-column call chain `!$omp declare target` and replaced the ragged locals with an `NK_GPU_MAX` + fixed size (§7.3) — so the two hard design questions have candidate answers (de-polymorphize by + *using the OM4 select-case path* rather than rewriting `Recon1d`; fix ragged sizing with a max-`nk` + pad). *Recommended approach:* validate/adopt that branch's OM4-path direction (settle the + `NK_GPU_MAX` occupancy cost first — §7.3, KNOWLEDGE.md §9), then add the missing driving device loop at + `MOM_ALE.F90:745` (`do concurrent (j,i)` over columns), then bitwise-validate against CPU with + `MOM_checksums`. This is still the highest-risk item and should be treated as a research problem, + but it is no longer a blank slate. +7. **`MOM_regridding.F90`** (grid generation) — simpler than remapping (integer `select case` + dispatch, no polymorphism found), likely portable with the standard template once someone + starts; currently zero in-flight work. +8. **`MOM_thickness_diffuse.F90`** (lateral, GM/isopycnal height diffusion) — technically lateral + not vertical, but bundled here because it's explicitly grouped with this porting frontier in + the architecture doc. Furthest along of any item in this document (real `do concurrent` + + `DO_LOCALITY` in-tree on `port/thickness_diffuse`), but drags in the density-integral/EOS + subsystem (§4) — expect a wide-footprint PR when it lands. +9. **`MOM_mixed_layer_restrat.F90`** (Bodner MLE restratification) — zero on mainline; one + contrasting **naive** in-flight port (`bodner-naive-port`) exists per the architecture doc, + useful as a second data point alongside `epbl-3d` for the "naive vs. k-blocked" trade-off + discussion, but not surveyed in depth here (out of this document's primary file list). + +### Tier 3 — peripheral / infrequent / not on the hot loop +10. **`MOM_diag_mediator.F90`** and diagnostic-only device-buffer-sync shims (the `find_eta` + pattern in §1) — real but low-value work; already tracked in `12-diagnostics-io.md`. +11. Regridding coordinate generators (`coord_rho.F90`, `coord_hycom.F90`, etc.) — called once per + remap step, but simple compared to the reconstruction machinery itself; will likely port + "for free" once `MOM_regridding.F90`/`MOM_remapping.F90` are handled. + +**Bottom line:** the diabatic column-physics kernels (Tier 1) are individually tractable — each +one needs the same `!$omp target teams loop` + `do concurrent(i)` + serial-`do k` shape already +proven twice (`MOM_vert_friction.F90` merged, `MOM_diabatic_aux.F90::find_uv_at_h` merged) and +partially rehearsed a third time (`epbl-3d`, naive variant). The real open problem is **ALE +remapping**: its two design questions (polymorphic per-column dispatch + ragged column sizes) are the +hardest in this survey, but — correcting the earlier draft — they are **not untouched**: +`jorge/diagnostics_port` has a preparatory device-enablement pass (OM4 path + `declare target` + +`NK_GPU_MAX`) that answers both in candidate form (§7.3). What remains is the driving device loop over +columns and bitwise validation, not a from-scratch redesign. + +--- + +## 9. Branches referenced in this document (for follow-up) + +| Branch | Role | +|---|---| +| `remotes/edoyango/port-set_diffusivity` | set_diffusivity j-blocking + array reordering (pre-device) | +| `remotes/edoyango/port/set_diffusivity` | superset of above, +BBL tile indices, TODO markers | +| `remotes/edoyango/set_diffusivity-kjiarrs` | earlier checkpoint of the same lineage | +| `remotes/edoyango/port/thickness_diffuse` | furthest-along lateral/vertical-mixing port, real `do concurrent`+`DO_LOCALITY` | +| `remotes/origin/epbl-3d` (+ `epbl-debug*`) | naive whole-column `do concurrent` port of EPBL, "100x" claim, workaround catalogue | +| `remotes/edoyango/port-set_viscous_BBL-tile`, `port-set_viscous_ML-tile`, `tile-vertvisc-coef` | adjacent tiling work on already-merged viscosity/friction modules, same idiom | +| `remotes/origin/find-eta-gpu`, `find-eta-merge`, `find-eta-with-CS` | ported `find_eta`/`MOM_interface_heights.F90`, consumed by the 3-line `MOM_ALE.F90`/`MOM_diabatic_driver.F90`/`MOM_thickness_diffuse.F90` diffs on mainline | +| `remotes/origin/jorge/diagnostics_port` | **the one preparatory remap-offload attempt**: `declare target` on the OM4 `remapping_core_h` chain + `NK_GPU_MAX` fixed sizing (§7.3); not yet device-driven | +| `remotes/JorgeG94/jorge/use_dp_as_real` | `-r8` → explicit `real64` kinds across ALE, plausible device-code prerequisite | +| `remotes/edoyango/submod-conversion` | mechanical, whole-codebase module→submodule split (compile-speed, not GPU) | +| `remotes/origin/bodner-naive-port` | naive-port contrast case for `MOM_mixed_layer_restrat.F90` + density integrals (architecture doc §6.2) | + +No file outside `docs/gpu-knowledge/14-vertical-physics-ale-status.md` was created; no build or +run was performed for this survey. + +--- + +## Verification notes + +Independently verified against source + git (branch `dev/gpu`, baseline `dev-gfdl`); no code built or run. + +**Confirmed exactly:** +- §1 numstat table — every row reproduced via per-file `git diff --numstat dev-gfdl...dev/gpu` + (0/0 for the six untouched modules; 3/0 for `MOM_diabatic_driver`/`MOM_ALE`/`MOM_thickness_diffuse`; + 2/1 `kappa_shear`; 13/12 `MOM_diabatic_aux`; 775/184 vertvisc; 501/384 set_viscosity). +- The three "3-line" diffs are byte-for-byte the `find_eta` device-buffer-sync pattern quoted; + `kappa_shear` is a CPU `!$OMP ... shared()` data-race fix (not a port). +- `find_uv_at_h` port pattern (§2.4): `!$omp target teams loop` over `j` + `do concurrent (i)` + + serial `do k` recurrence, `map(alloc/release)` on tridiagonal temporaries — exact. +- Diabatic dispatch structure and all kernel call-site line numbers (§2.1 table) — exact. +- `port-set_diffusivity`: numstat (204/2, 130/110, 693/584), commit log, and **zero** added + `omp target`/`do concurrent`/`DO_LOCALITY` lines (preparatory-only) — confirmed. +- `port/thickness_diffuse`: 18-file footprint (incl. `pkg/CVMix-src` submodule bump → "17 source + files") and real device directives (135 added `do concurrent`/`DO_LOCALITY` lines) — confirmed. +- `epbl-3d`: `pure ePBL_column[_3d]`, `do concurrent (j,i)`, and all three commit-message claims + ("100x…~2ms/step", "Replace auto array size (nk=75)" → literal `75`, "Overly aggressive + inlining") — confirmed verbatim. +- ALE: `MOM_ALE.F90:745-748` host loop; `Recon1d` abstract type + 10 deferred procedures + (`Recon1d_type.F90:16`); `class(Recon1d), pointer` (`:83`) + `REMAPPING_VIA_CLASS` dispatch + (`:273-300`); `MOM_regridding.F90:1281` integer `select case`, no polymorphism — all confirmed. +- Tier prioritization is consistent with the arch-doc §4 call tree (diabatic column kernels run + every thermo step; ALE remap every step in ALE mode; thickness_diffuse/MLE are config-conditional). + +**Corrected:** +- §5 `epbl-3d` numstat block **understated the footprint** — the branch touches 6 files + (`MOM_interface_heights.F90`, `smod.mk`, `MOM_wave_interface.F90` + `_smod`), not just the two + EPBL files. Fixed, with note that `MOM_wave_interface` is submodulified for device reachability. +- §7.3 / §8 **the central "no branch attempts remap offload / zero in-flight work, preparatory or + otherwise" claim was wrong.** `remotes/origin/jorge/diagnostics_port` (`MOM_remapping.F90` 82/62, + not 78/61) marks the entire OM4 per-column chain `!$omp declare target` and adds + `NK_GPU_MAX = 500` fixed sizing (`:47`,`:1275-1277`) — a genuine preparatory device-enablement + attempt that sidesteps `Recon1d` polymorphism via the OM4 select-case path. It stops short of a + driving device loop. Rewrote the §7.3 row, the §7.2 point-2/3 claims, the §7.3 conclusion, the §8 + Tier-2 item 6, and the bottom line accordingly; added it to §9. +- §7.2 point 3 listed `adjust_h_sub` as a live per-column call — it is **commented out** + (`MOM_remapping.F90:280`,`:809`,`:843`). Corrected; call list updated to the real OM4 chain. +- §2.1/§2.3: the kernels are dispatched from **three** routines (`diabatic_ALE_legacy` included), + not two — clarified (set_diffusivity sites 694/697 live in the legacy path). +- Minor numstat drift on `cmake/amd-flang` `MOM_remapping.F90` (36/34, doc had 32/33) — updated. + +**Enhancements:** Tier-1 targets now carry per-item recommended approach (merged precedent: +`find_uv_at_h`/vertvisc teams-loop for tridiagonals, `epbl-3d` for whole-column, EOS `_loc` + +`isopycnal_slopes` prerequisite chain for `set_diffusivity`), known hazards (pointer `tv%T`/`tv%S` +and restart-target `visc%Kd_*` members, diag `id_*>0` guards), CVMix external-library inlining risk, +and an explicit dependency ordering (EOS `_loc` → N²/density → set_diffusivity → KPP/EPBL → +kappa_shear). ALE remap now points at the concrete `diagnostics_port` groundwork. + +**Confidence:** High. Every numstat, line number, commit hash, and code excerpt was checked directly +against the tree; the one material correction (diagnostics_port remap groundwork) was verified by +reading the actual `declare target`/`NK_GPU_MAX` hunks on that remote branch. diff --git a/knowledge/gpu-knowledge/nvfortran-automatic-array-bug.md b/knowledge/gpu-knowledge/nvfortran-automatic-array-bug.md new file mode 100644 index 0000000..ee6cab1 --- /dev/null +++ b/knowledge/gpu-knowledge/nvfortran-automatic-array-bug.md @@ -0,0 +1,85 @@ +# nvfortran: automatic arrays in device-called procedures + +**One line:** nvfortran cannot allocate an **automatic array on the device stack** when its size +comes from a runtime expression that is **not a dummy argument**, inside a procedure **called from** +a device region. + +## Symptom + +Device compile failure (or a forced fixed-size workaround) on a local array like: + +```fortran +real, dimension(SZK_(GV)+1) :: dz_col ! SZK_(GV) -> GV%ke : a derived-type component +``` + +in a routine invoked from inside a `!$omp target`/`do concurrent` region. The compiler needs a +per-thread stack size it cannot know: `GV%ke` is a runtime value reached through a derived type, +and there is no device-side dynamic stack to fall back on. + +## What it is *not* + +It is **not** "automatic arrays can't be privatized", and it is **not** specific to +`do concurrent`. This is merged on `dev/gpu` and bitwise-gated, and it privatizes an automatic +array just fine: + +```fortran +subroutine vertvisc ! MOM_vert_friction.F90 + real :: c1(SZK_(GV)) ! :575 automatic, sized GV%ke + + !$omp target teams loop collapse(2) & ! :737 WORKS + !$omp private(b1, c1, d1, Ray, b_denom_1) +``` + +## The actual distinction + +| Shape | Works? | Why | +|---|---|---| +| Automatic array in the directive's **own routine**, listed in `private(...)` | **Yes** | the host sizes it at routine entry; the compiler emits N per-thread copies | +| Automatic array **local to a callee** invoked from a device region, sized from non-dummy data | **No** | would need device-side dynamic stack allocation | + +## Evidence + +`05c74b56b` — Marshall Ward, 2026-06-03, *"ePBL: Replace auto array size (nk=75)"*, on the +epbl-3d branch (`MOM_energetic_PBL_smod.F90`): + +```fortran +! in ePBL_column_3d -- CALLED from the device loop at :528 +- real, dimension(SZK_(GV)+1) :: ... ++ real, dimension(75+1) :: ... ! hardcoded to escape the bug +``` + +The failing construct there was `!$omp target loop private(SpV_dt)` in a `module subroutine` — +**neither `pure` nor `do concurrent`**. + +## Workarounds, best to worst + +1. **Size the automatic from a dummy argument** (`size(h,3)`, or an `nz` dummy). Then nvfortran can + size it. *Untested — this is the cheap experiment to run before doing anything below.* +2. **Hoist the array to the caller and `private()` it** on the directive — the `vert_friction:737` + pattern above. Known to work. +3. **Compile-time `parameter`** (`nk=75`, `NK_GPU_MAX=500`). Works, but over-allocates every column + to the max: ~6.7x waste at a realistic `GV%ke≈75`, which will spill and crush occupancy. + Parameterize before merging anything that relies on it. +4. **Static memory mode** sidesteps it entirely — `MOM_memory_macros.h:86` makes `SZK_(G)` expand to + `NK_`, a compile-time constant, versus `:172`'s `G%ke` in dynamic mode. **So this bug only bites + dynamic-memory builds.** Check which one you're on before chasing it. + +## Status + +- **Verified from source:** the `vert_friction:737` counter-example (merged + checksum-gated) and + the `05c74b56b` diff. No build/run was done. +- **Inferred, untested:** that sizing from a dummy argument fixes it (workaround 1). +- **Unclear:** which nvfortran versions are affected. Our evidence is 25.x; the + `porting-mom6-skill` skill claims 26.5 but may have inherited rather than re-tested the claim. + +## Note for the two existing write-ups + +Both are misleading and should be corrected against the table above: + +- **`porting-mom6-skill/SKILL.md`** says `do concurrent` cannot privatize an automatic array and to + *"fall back to `!$omp target teams loop`"*. But the failing construct was `target loop`, not DC — + and the suggested fallback is exactly what already works (`vert_friction:737`). It would send an + agent to swap constructs, which either succeeds for the wrong reason or hits the same wall. +- **`KNOWLEDGE.md` §5 row 21** says *"in `pure`/DC procedures"* — but the evidence commit is neither. + (§9, "`NK_GPU_MAX=500` sizing", gets it right: "non-dummy-sized automatics", and already + recommends workaround 1.) diff --git a/skills/gpu-data-residency/SKILL.md b/skills/gpu-data-residency/SKILL.md new file mode 100644 index 0000000..2b468c9 --- /dev/null +++ b/skills/gpu-data-residency/SKILL.md @@ -0,0 +1,179 @@ +--- +name: gpu-data-residency +description: Decide where `!$omp target enter data map(...)` belongs for a MOM6 array, choose `map(to:)` vs `map(alloc:)`, and find every place a device-resident array must be copied back to the host (`target update from`) or refreshed on device (`target update to`). Use when porting a module to GPU, adding a device-resident array, auditing a port for missing transfers or unbalanced enter/exit data, or debugging a checksum mismatch, answers-differ-by-GPU-count, or stale-host symptom. Triggers on "where do I map this", "enter data", "exit data", "update from", "copy back", "needed on the host", "residency", "stale host", "map(to) vs map(alloc)", "missing transfer", "map balance". +--- + +# GPU data residency: where to map, and where to copy back + +Answers two questions for a given array (or every array in a module): + +1. **Where does its `enter data`/`exit data` go, and with which map kind?** +2. **Where must it be copied back** (`update from`) **or refreshed** (`update to`) **because a host-only consumer touches it?** + +Read `knowledge/KNOWLEDGE.md` §3 Step 5 (mapping lifecycle) and §3 Step 8 (transfer discipline) if not already in context. +Depth: `knowledge/gpu-knowledge/03-openmp-mapping.md`, `12-diagnostics-io.md`, `02-pointer-usage.md`. + +## The engine: every mapped array is a two-copy shadow state + +Track **coherence between the host copy and the device copy** as you walk the code in execution +order. Almost every mapping bug in this tree is a state-machine violation. + +| State | Host copy | Device copy | +|---|---|---| +| `UNMAPPED` | authoritative | does not exist | +| `SYNCED` | valid | valid | +| `HOST_FRESH` | authoritative | **stale / garbage** | +| `DEV_FRESH` | **stale / garbage** | authoritative | + +| Event | Resulting state | Note | +|---|---|---| +| `enter data map(to: x)` | `SYNCED` | **Only if not already present.** On an already-present object this is a refcount bump and **copies nothing** — state unchanged (§8, "a `map(to:)` on an already-present object does not refresh device contents"). | +| `enter data map(alloc: x)` | `HOST_FRESH` | Device side is garbage. Legal only if the next device touch is a *write*. | +| host write | `HOST_FRESH` | includes `!$OMP parallel do` loops — those are **host** CPU threads | +| device write | `DEV_FRESH` | | +| **host read while `DEV_FRESH`** | **BUG** | insert `!$omp target update from(x)` before it | +| **device read while `HOST_FRESH`** | **BUG** | insert `!$omp target update to(x)` before it | +| `update from(x)` | `SYNCED` | device → host | +| `update to(x)` | `SYNCED` | host → device | +| `exit data map(from: x)` | `UNMAPPED`, host valid | copies back | +| `exit data map(delete:/release: x)` | `UNMAPPED`, host **as it was** | **neither kind copies back** | + +The machine tracks *coherence*, not *initialization*: `HOST_FRESH` on a freshly-`map(alloc:)`'d +local automatic array means "host is authoritative and holds garbage". Both reads are still wrong; +flag them separately. + +Two rules the mechanical walk will not derive on its own: + +- **`delete` vs `release`** — `delete` forces the refcount to **zero**, destroying any *outer* + persistent mapping of the same object. Use `release` for scoped/per-call teardown; `delete` only + in the `*_end` that mirrors the owning `enter data`. Never `map(delete:)` an object your scope + does not own (`vertvisc`'s `map(delete: ADp)` silently kills `initialize_MOM`'s map — §8, "the `ADp` mapping lifecycle"). +- **Re-mapping never refreshes.** If host scalars/descriptors changed after the first map, the only + refresh is `target update to(...)`. Never "re-map to refresh"; never re-`enter data` a parent + struct after its members are attached (`c82e1254a`). + +## Procedure + +### Step 1 — Scope and inventory + +Pick the array(s). For each, get every touch site interleaved with every region marker, in line +order: + +```bash +scripts/residency-scan.sh +``` + +It tags each line `MAP` / `XFER` / `DEV-REGION` / `DEV-HALO` / `HOST-THREADS` / `HOST-SINK` / +`TOUCH`. It is a *reading aid*, not an oracle — it tags the lines that open regions, and you still +have to read the code to see which touches fall inside them. + +Also establish the array's identity, which fixes where the map goes: + +| Kind | `enter data` site | `exit data` site | +|---|---|---| +| CS member (`ALLOCABLE_`) | in `_init`, next to `ALLOC_`, after `... = 0.0` | in `_end`, next to `DEALLOC_`, mirrored member-by-member, `delete` | +| Subroutine-scope scratch | at routine entry, `map(alloc:)` | at return, `release` (early-release once last use passes is fine) | +| Dummy argument | **not here** — the caller owns it; verify residency at every call site | — | +| Pointer member | `map(to:)` **never `alloc`**, guarded `if (associated(x))` | mirrored, same guard | + +### Step 2 — Classify every touch HOST or DEVICE + +This is where the analysis is won or lost. Reason about the **GPU build** +(`__NVCOMPILER_OPENMP_GPU`). + +| Marker | Verdict | +|---|---| +| `do concurrent (...)` | **DEVICE** — the default compute idiom | +| `!$omp target teams` / `!$omp target ... loop` | **DEVICE** | +| `!$OMP parallel do` / `!$omp parallel` | **HOST** — CPU threads. *The single most common misread.* | +| plain `do` loop | **HOST** | +| plain `do` loop **inside** a `target teams loop` / DC | **DEVICE** (serial-k columns) | +| call to a `pure`/`elemental`/`declare target` helper from a device region | **DEVICE** | +| any other `call` | **HOST** unless proven otherwise | + +Then check the touch against the **host-boundary catalogue** — the calls that are host-only and +therefore force a copy-back. See `references/host-boundaries.md` for the full list with its +verification commands. The load-bearing ones: + +- `post_data` and the whole diag mediator; `hchksum`/`uvchksum`/... ; `save_restart` +- `pass_var`/`pass_vector` (no `omp_offload` argument exists); `start_group_pass`/`complete_group_pass` +- `do_group_pass(..., omp_offload=.true.)` is **DEVICE** — no transfer. Without the flag: HOST. +- any call into an untouched module (diabatic stack, ALE remap/regrid, restart — `knowledge/KNOWLEDGE.md` §2.4) + +### Step 3 — Walk the ledger + +In execution order, one row per event. **Branches matter more than anything else here**: a transfer +inside `if (cond)` does not dominate a read outside it. When a device write and a host read sit in +sibling branches, write down the predicate that reaches the read without the transfer — that +conjunction *is* the bug report. + +| # | Line | Event | Host/Dev | State after | Verdict | +|---|---|---|---|---|---| +| 1 | `:209` | `map(alloc: khdt_x)` | — | `HOST_FRESH` (garbage) | ok | +| 2 | `:291` | write in `do concurrent` | DEV | `DEV_FRESH` | ok | +| 3 | `:394` | read in plain `do` | HOST | — | **BUG: needs `update from`** | + +### Step 4 — Emit directives + +Map kind, from the *first device touch* and who else reads the contents: + +- first device touch is a **read**, or any host-set scalar/pointer descriptor is read on device + → **`map(to:)`**. This includes every struct whose `associated()` state feeds device control flow + (`map(alloc: Reg, Reg%Tr(:))` was the multi-GPU answer-change bug, `a774eb331`). +- first device touch is a **write**, pure workspace → **`map(alloc:)`**. +- CS shells → `map(alloc:)`, mapped **once**, before the child `_init`. + +Transfer placement: + +- Put the transfer at the **producer**, immediately upstream of the consumer. A transfer of a + *different* array does not cover yours (`b29b27150`). +- **Decouple transfer from post**: one `update from` covering all consumers, guarded + `if (CS%debug .or. CS%id_a>0 .or. CS%id_b>0)`, then the individual `if (id>0) call post_data(...)` + (`MOM_diagnostics.F90:1825-1827`). +- Bracket an unavoidable host-only detour both ways: `from(...)` before, `to(...)` after + (ALE at `MOM.F90:1036/1038`). +- At coarse sync points a blanket transfer is the codebase default (`MOM.F90:1091`) — match it + unless profiling shows a stall. +- Guard with the matching intrinsic: `if (associated(x))` for pointers, `if (allocated(x))` for + allocatables — never mixed. Never `map(...) if (present(optional))` inside a callee + (`2108e0eba`). + +### Step 5 — Lifecycle and balance checks + +Run these over the routine/module regardless of what the walk found: + +1. **Every `enter data` has a mirrored `exit data`** in the same scope (`15ca2a25f` leaked + `b_denom_1`). +2. **No `delete` on an object this scope does not own** (§8, "the `ADp` mapping lifecycle"). +3. **No copy-back expected from `delete`/`release`** — if the host needs the value, an `update from` + or `map(from:)` must precede it. +4. **Parent mapped exactly once**, members attached after, refreshed with `update to(parent)`. +5. **Restart-registered, device-mutated fields** have a dominating `update from` before + `save_restart` (currently latent-only — §8, "restart staleness is latent, not live"; do not let your port break it). +6. **Arrays-of-structs are not mapped element-by-element** on a hot path (`1865612de`). + +### Step 6 — Report + +Lead with the verdict. For each finding give: array, the two sites (device write → host read), the +**predicate that reaches the bad read**, the symptom it would produce, and the one-line fix. Then +the proposed directive set. Separate **confirmed** (you read both sites and the branch structure) +from **suspected** (needs a run). + +## Symptoms this analysis explains + +| Symptom | Likely residency cause | +|---|---| +| Checksum "mismatch" that isn't reproducible arithmetic | host-only checksum read a stale host copy — missing `update from` (`b29b27150`) | +| Answers differ **by GPU count** / run-to-run, control flow | `map(alloc:)` on a struct whose host-set scalars or `associated()` are read on device (`a774eb331`) | +| Device "addressing error" after init; `associated()` misbehaves | parent re-`enter data`'d after members attached (`c82e1254a`) | +| Correct on 1 GPU, wrong on N | missing `reduce`, or `alloc`-vs-`to` — latent until multi-device | +| Silent garbage in a host diagnostic only in some configs | transfer guarded by a *different* predicate than the read (the `khdt_x` shape — see `references/worked-example.md`) | +| GPU time dominated by attach/detach | array-of-structs mapped per element (`1865612de`) | +| Unexplained per-statement traffic | array touched in a device region with no explicit map at all (`bc05a6a89`) | + +## Verification gate + +A residency fix is a correctness change: it is subject to the same gate as any port +(`knowledge/KNOWLEDGE.md` §3 Step 9) — bit-identical `MOM_checksums` field checksums plus EFP +`write_energy`, and a **≥2-GPU run**, because the `alloc`-vs-`to` class of bug is latent on one +device. Never accept a nonzero diff as rounding. diff --git a/skills/gpu-data-residency/references/host-boundaries.md b/skills/gpu-data-residency/references/host-boundaries.md new file mode 100644 index 0000000..1de624a --- /dev/null +++ b/skills/gpu-data-residency/references/host-boundaries.md @@ -0,0 +1,88 @@ +# The host-boundary catalogue + +Every entry is a **host-only consumer**: if it touches an array that is `DEV_FRESH`, a +`!$omp target update from()` must dominate it. If it *writes* an array the device +later reads, an `update to(...)` must follow it. + +Verified against `dev/gpu` HEAD on 2026-07-14. Each row carries the command that re-establishes it — +re-run them if the tree has moved, rather than trusting this table. + +## Host-only by module (zero device directives in the whole file) + +| Consumer | Why host-only | Re-verify | +|---|---|---| +| `post_data`, `register_diag_field`, the entire diag mediator | 0 `omp target` directives | `grep -c "omp target" src/framework/MOM_diag_mediator.F90` → 0 | +| `hchksum`, `uchksum`, `vchksum`, `uvchksum`, `Bchksum`, `chksum`, `MOM_tracer_chksum` | 0 `omp target` directives | `grep -c "omp target" src/framework/MOM_checksums.F90` → 0 | +| `save_restart`, `save_MOM_restart`, `register_restart_field` | 0 `omp target` directives | `grep -c "omp target" src/framework/MOM_restart.F90` → 0 | + +`MOM_restart.F90` does **no transfer of its own**. Restart staleness is currently latent, not live, +only because the sync-point blanket `update from(u, v, h, CS%uhtr, CS%vhtr)` at `MOM.F90:1091` runs +under the same condition the driver writes restarts under (`knowledge/KNOWLEDGE.md` §8, "restart staleness is latent, not live"). Any +*newly* device-resident restart-registered field you add must be added to a dominating +`update from` before `save_restart`. + +## Halo exchange — the one place the verdict flips + +| Call | Verdict | Why | +|---|---|---| +| `do_group_pass(group, dom, omp_offload=.true.)` | **DEVICE** — no transfer needed | forwards to FMS `mpp_do_group_update`, which device-packs halos and posts `MPI_ISEND`/`IRECV` under `!$omp target data use_device_ptr(...)` — real CUDA-aware MPI on device pointers, with **no host-staging fallback** (§8, "FMS `omp_offload` is a genuine device path") | +| `do_group_pass(...)` **without** the flag | HOST | bracket it `update from` / `update to` | +| `pass_var`, `pass_vector` | **HOST** — always | no `omp_offload` argument exists on these entry points | +| `start_group_pass` / `complete_group_pass` | **HOST** — always | the nonblocking path hardcodes `use_device_ptr = .false. ! placeholder` in FMS | + +Re-verify the `pass_var`/`pass_vector` claim (the argument lists must contain no `omp_offload`): + +```bash +sed -n '173,175p;662,664p' config_src/infra/FMS2/MOM_domain_infra.F90 +grep -n "omp_offload" config_src/infra/FMS2/MOM_domain_infra.F90 +``` + +⚠ 14 of the 26 `omp_offload=.true.` sites are gated behind `if (G%nonblocking_updates)` and revert +to the host-staged branch when it is on — so the *same call site* is a host boundary or not +depending on a runtime parameter (`knowledge/KNOWLEDGE.md` §9, "`NONBLOCKING_UPDATES` policy", open). If your analysis depends on a +gated site, say which branch you assumed. + +## Host-only by call type + +| Consumer | Note | +|---|---| +| `max_across_PEs`, `min_across_PEs`, `sum_across_PEs` | host MPI | +| `reproducing_sum`, `reproducing_sum_EFP` | GPU-aware internally (`8593a732a`), but confirm the *inputs* it reads are resident; never hand-roll a float sum | +| `write_energy` | brackets its own `to(tv%S, tv%T)` at `MOM_sum_output.F90:762` | +| `MOM_error`, any I/O, any `print`/`write` | host | +| `get_param` / `param_file` | host, init-time | + +## Host-only by module status (the untouched list) + +Calling into any of these is a host boundary — they have **0 diff vs `dev-gfdl`** and no device +awareness (`knowledge/KNOWLEDGE.md` §2.4): + +`MOM_set_diffusivity.F90`, `MOM_CVMix_KPP.F90`, `MOM_energetic_PBL.F90`, +`MOM_mixed_layer_restrat.F90`, `MOM_regridding.F90`, `MOM_remapping.F90`, +`MOM_diag_mediator.F90`, `MOM_restart.F90` — i.e. the whole **diabatic stack**, **ALE remap +machinery**, and **diagnostics/restart IO**. + +The two structural brackets that exist today because of this: + +- ALE remap: `update from(u,v,h)` at `MOM.F90:1036` → host-only remap → `update to(u,v,h)` at `:1038` +- diabatic: host-only throughout; `:1827` brackets it + +Re-verify the untouched list before relying on it (branches land): + +```bash +git diff --stat dev-gfdl...dev/gpu -- src/parameterizations/vertical/MOM_set_diffusivity.F90 +``` + +## The classification traps + +1. **`!$OMP parallel do` is HOST.** CPU threads, not device. It looks like an OpenMP offload + directive and is not. This is the most common misread in this codebase, and it is *everywhere* + in the same files as real `target` directives (`MOM_tracer_hor_diff.F90:297-382` has eight of + them interleaved with device regions). +2. **`do concurrent` is DEVICE** on the GPU build — the default compute idiom (698 uses). +3. **A commented-out directive is not a directive.** `!!$omp target update from(Shear_mag)` at + `MOM_hor_visc.F90:1656-1658` and `!**!$omp target update to(...)` at `MOM.F90:867` are disabled. + Do not count them as transfers. +4. **A transfer of a *different* array does not cover yours** (`b29b27150`). +5. **A transfer under a different predicate than the read does not cover it.** This is the + highest-yield bug shape — see `worked-example.md`. diff --git a/skills/gpu-data-residency/references/worked-example.md b/skills/gpu-data-residency/references/worked-example.md new file mode 100644 index 0000000..d2592d8 --- /dev/null +++ b/skills/gpu-data-residency/references/worked-example.md @@ -0,0 +1,118 @@ +# Worked example: `khdt_x` in `tracer_hordiff` + +`src/tracer/MOM_tracer_hor_diff.F90`, subroutine `tracer_hordiff` (from `:122`). This one array +exercises every state transition, every classification trap, and the highest-yield bug shape. Work +through it once before running the procedure on something new. + +Raw material: + +```bash +skills/gpu-data-residency/scripts/residency-scan.sh \ + src/tracer/MOM_tracer_hor_diff.F90 khdt_x +``` + +## Step 1 — Identity + +`khdt_x` is declared at `:162` as **subroutine-scope scratch** (a local automatic, `Khtr*dt` times +face width). So its map belongs at routine entry / return, not in a `_init`/`_end`. It is: + +- `:209` `enter data map(alloc: khdt_x, khdt_y, kh_u, kh_v)` — entry, `alloc` +- `:723` `exit data map(release: khdt_x, khdt_y, Kh_u, Kh_v)` — return, `release` not `delete` ✓ + +Both correct: `alloc` because the first device touch is a write, `release` because this scope's map +is per-call and `delete` would zero the refcount on anything an outer scope had mapped. + +## Step 2 — The ledger + +Four mutually exclusive branches produce `khdt_x`. Walk each to the host read at `:394`. + +| # | Line | Event | H/D | State after | Verdict | +|---|---|---|---|---|---| +| 1 | `:209` | `map(alloc: khdt_x)` | — | `HOST_FRESH` (garbage) | ok | +| | | **Branch A — `do_online .and. use_VarMix`** | | | | +| 2 | `:290` | `do concurrent (j,I)`, `khdt_x(I,j) = ...` | **DEV** | `DEV_FRESH` | ok | +| 3 | `:338` | `if (CS%max_diff_CFL > 0.0)` | — | — | ⚠ predicate | +| 4 | `:339` | `update from(khdt_x, ...)` | — | `SYNCED` | **only inside #3** | +| 5 | `:341` | `!$OMP parallel do` limiter, reads+writes `khdt_x` | **HOST** | `HOST_FRESH` | ok (dominated by #4) | +| 6 | `:374` | `update to(khdt_x, ...)` | — | `SYNCED` | ok | +| | | **Branch B — `Resoln_scaled`** | | | | +| 7 | `:297` | `!$OMP parallel do`, writes `khdt_x` | **HOST** | `HOST_FRESH` | ok | +| 8 | `:309` | `update to(khdt_x, ...)` | — | `SYNCED` | ok | +| | | **Branch C — constant diffusivity** | | | | +| 9 | `:312-333` | `!$OMP parallel do`, writes `khdt_x` | **HOST** | `HOST_FRESH` | ok | +| 10 | `:335` | `update to(khdt_x, ...)` | — | `SYNCED` | ok | +| | | **Branch D — `.not. do_online`** | | | | +| 11 | `:378` | `!$OMP parallel do`, `khdt_x = read_khdt_x` | **HOST** | `HOST_FRESH` | ok | +| 12 | `:386` | `call pass_vector(khdt_x, khdt_y, ...)` | **HOST-SINK** | `HOST_FRESH` | ok — host array, host halo | +| 13 | `:387` | `update to(khdt_x, ...)` | — | `SYNCED` | ok — refresh after host halo | +| | | **Join** | | | | +| 14 | `:390` | `if (CS%check_diffusive_CFL)` | — | — | ⚠ predicate | +| 15 | `:394` | `CFL(i,j) = 2.0*((khdt_x(I-1,j) + ...` in a plain `do` | **HOST** | — | **BUG on branch A** | +| 16 | `:438` | `update from(khdt_x, khdt_y)` under `use_hor_bnd_diffusion` | — | `SYNCED` | ok — dominates `:441+` | +| 17 | `:722` | `update from(khdt_x, khdt_y) if (CS%debug .or. CS%id_khdt_x>0 .or. ...)` | — | `SYNCED` | ok — the decoupled-transfer idiom | +| 18 | `:726/:731` | `uvchksum` / `post_data(CS%id_khdt_x, ...)` | **HOST-SINK** | — | ok (dominated by #17, guard matches) | + +## Step 3 — The finding + +Rows 2 → 15. On **branch A only**, `khdt_x` is written on device and the host copy is never +written at all. The copy-back at `:339` is guarded by `CS%max_diff_CFL > 0.0`; the host read at +`:394` is guarded by `CS%check_diffusive_CFL`. **These are independent runtime parameters** — the +transfer's predicate does not dominate the read's: + +``` +CHECK_DIFFUSIVE_CFL default .false. (:1745) +MAX_TR_DIFFUSION_CFL default -1.0 (:1750) +``` + +Reaching predicate for the stale read: + +``` +do_online .and. use_VarMix .and. .not.(CS%max_diff_CFL > 0.0) .and. CS%check_diffusive_CFL +``` + +i.e. `CHECK_DIFFUSIVE_CFL = True` with the **default** `MAX_TR_DIFFUSION_CFL`, and variable mixing +on — a natural configuration ("iterate to respect the CFL limit" without local diffusivity +limiting), not an exotic one. + +Consequence: `max_CFL` is computed from an uninitialized host `khdt_x`, `max_across_PEs` at `:399` +spreads it, and `num_itts = max(1, ceiling(max_CFL - ...))` at `:401` picks a garbage iteration +count for the tracer diffusion — wrong answers, or a huge `num_itts` and a hang, depending on what +the uninitialized memory holds. `CFL` is also posted as a diagnostic at `:403`. + +Note the symptom this would *not* produce: it is invisible to a default-config checksum run, and +invisible on branches B/C/D. That is why the ledger is worth writing down rather than eyeballed. + +One-line fix — move the copy-back to the consumer's predicate: + +```fortran + if (CS%check_diffusive_CFL) then + !$omp target update from(khdt_x, khdt_y) ! <-- add + if (CS%show_call_tree) call callTree_waypoint("Checking diffusive CFL (tracer_hordiff)") +``` + +(Redundant with `:339` when both predicates hold — a second `update from` of a `SYNCED` array is a +wasted copy, not a bug. If that matters, guard it `.and. .not.(CS%max_diff_CFL > 0.0)`, at the cost +of a predicate that has to be maintained in lockstep with `:338`. Prefer the simple version.) + +## What this example teaches + +1. **Branches are the analysis.** A device write and a host read in the same routine with a + transfer *somewhere* between them proves nothing. Only a transfer whose predicate is implied by + the read's predicate dominates it. +2. **`!$OMP parallel do` is a host loop.** Eight of them here, interleaved with `do concurrent` + device regions in the same `if/elseif` chain. Rows 5, 7, 9, 11 are host; row 2 is device. Read + the directive, not the indentation. +3. **`map(alloc:)` means the host copy is garbage** until someone writes it. Branch A never does. +4. **The right idiom is right there in the same file**, at `:722` — one guarded `update from` + whose condition is the disjunction of every consumer's condition + (`CS%debug .or. CS%id_khdt_x>0 .or. CS%id_khdt_y>0`), then the individual consumers each behind + their own guard. Copy that shape. +5. `pass_vector` at `:386` is a **host** halo exchange on a host-fresh array, correctly followed by + `update to`. Compare `do_group_pass(..., omp_offload=.true.)`, which needs no transfer at all. + +## Status + +Reported 2026-07-14, found by running this skill's procedure while writing it. Source-only analysis +— not yet confirmed by a run. Confirming it needs a `CHECK_DIFFUSIVE_CFL=True` + +`MAX_TR_DIFFUSION_CFL=-1` + VarMix case; the cheap tell is a nondeterministic `num_itts` / +`CFL` diagnostic on GPU vs CPU. diff --git a/skills/gpu-data-residency/scripts/residency-scan.sh b/skills/gpu-data-residency/scripts/residency-scan.sh new file mode 100755 index 0000000..2913d65 --- /dev/null +++ b/skills/gpu-data-residency/scripts/residency-scan.sh @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +# residency-scan.sh [more-array-names...] +# +# Prints, in line order, every touch of the named array(s) interleaved with every marker that +# opens a device region, a host-threaded region, a map/transfer directive, or a host-only sink. +# That interleaving is the raw material for the residency ledger (SKILL.md Step 3). +# +# Tags: +# MAP !$omp target enter/exit data +# XFER !$omp target update to/from +# DEV-REGION do concurrent, !$omp target teams/loop -> touches inside are DEVICE +# DEV-HALO do_group_pass(..., omp_offload=.true.) -> device-resident, no transfer needed +# HOST-THREADS !$OMP parallel do -> HOST CPU threads, NOT device +# HOST-SINK a host-only consumer (see references/host-boundaries.md) +# TOUCH a mention of the array +# +# This is a reading aid, not an oracle: it tags the lines that OPEN regions. You still have to read +# the code to see which touches fall inside which region, and Fortran is case-insensitive so name +# matching is too. + +set -euo pipefail + +if [ "$#" -lt 2 ]; then + echo "usage: $(basename "$0") [more...]" >&2 + exit 2 +fi + +file=$1; shift +if [ ! -r "$file" ]; then echo "cannot read: $file" >&2; exit 2; fi + +names=$(printf '%s|' "$@" | sed 's/|$//') + +awk -v names="$names" ' +BEGIN { names = tolower(names) } # Fortran is case-insensitive; source lines are lowercased below +{ + line = tolower($0) + tag = "" + + if (line ~ /!\$omp[ \t]+target[ \t]+(enter|exit)[ \t]+data/) tag = "MAP" + else if (line ~ /!\$omp[ \t]+target[ \t]+update/) tag = "XFER" + else if (line ~ /!\$omp[ \t]+target/) tag = "DEV-REGION" + else if (line ~ /do[ \t]+concurrent/) tag = "DEV-REGION" + else if (line ~ /!\$omp[ \t]+(parallel|do[ \t])/) tag = "HOST-THREADS" + else if (line ~ /call[ \t]+do_group_pass/) + tag = (line ~ /omp_offload[ \t]*=[ \t]*\.true\./) ? "DEV-HALO" : "HOST-SINK" + else if (line ~ /call[ \t]+(post_data|pass_var|pass_vector|start_group_pass|complete_group_pass|hchksum|uchksum|vchksum|uvchksum|bchksum|chksum|mom_tracer_chksum|save_restart|save_mom_restart|register_restart_field|max_across_pes|min_across_pes|sum_across_pes|global_area_mean|global_area_integral|reproducing_sum|write_energy|mom_error)/) + tag = "HOST-SINK" + + # A commented-out directive is not a directive. A live one has "!$omp" as the first + # non-blank text on the line; anything else in front of it (!!$omp, !**!$omp) disables it. + if (tag ~ /^(MAP|XFER|DEV-REGION|HOST-THREADS)$/) { + trimmed = line; sub(/^[ \t]+/, "", trimmed) + if (trimmed ~ /\$omp/ && trimmed !~ /^!\$omp/) tag = "DISABLED" + } + + touch = 0 + if (names != "" && line ~ ("(^|[^a-z0-9_%])(" names ")([^a-z0-9_]|$)")) touch = 1 + + if (tag == "" && !touch) next + if (touch) tag = (tag == "") ? "TOUCH" : tag "+TOUCH" + printf "%-18s %6d %s\n", tag, NR, $0 +}' "$file" diff --git a/skills/mom6-gpu-architect/SKILL.md b/skills/mom6-gpu-architect/SKILL.md new file mode 100644 index 0000000..9955336 --- /dev/null +++ b/skills/mom6-gpu-architect/SKILL.md @@ -0,0 +1,179 @@ +--- +name: mom6-gpu-architect +description: Map a MOM6 routine or module's structure and call graph, then design its GPU port — including whether the first step should be a bitwise-safe refactor (verbatim extraction into pure/elemental procedures) rather than attacking a 500-line monolithic loop directly. Use when picking up a new port target, asking "how should I port this", "what does this call", "is this portable as-is", "should I refactor first", "map out this module", or when sizing/sequencing porting work. Produces a port design; hand the mapping details to the gpu-data-residency skill and the mechanics to KNOWLEDGE.md §3. +--- + +# MOM6 GPU architect: map the routine, then design the port + +Produces a **port design** for one routine or module: a call map, a shape verdict, an explicit +refactor-or-port-in-place decision, a loop-form plan, and a sequenced work plan with gates. + +**This skill decides *what* to do and in *what order*. It does not restate *how*.** Hand off: + +| Question | Goes to | +|---|---| +| the porting mechanics (k-blocking, loop forms, verification) | `knowledge/KNOWLEDGE.md` §3, §4 | +| where to map arrays / where to copy back | the **`gpu-data-residency`** skill | +| device-callable helper rules | `knowledge/KNOWLEDGE.md` §3 Step 4, doc 08 | +| is this bitwise-safe? | `knowledge/KNOWLEDGE.md` §7.2, doc 07 | + +## Step 1 — Orient before measuring + +1. Read `knowledge/KNOWLEDGE.md` §6 (work queue) — **is this target already in flight?** §2.4 lists branches + carrying groundwork (j-blocking, `declare target` prep). Build on it; do not restart. Several + Tier-1 items have branches that already answer half the design. +2. Read §6's dependency ordering. Tier 1 is not a menu: **EOS `_loc` coverage → N²/density inputs + → `set_diffusivity` → KPP/EPBL → `kappa_shear`**. Designing `set_diffusivity` before the EOS + chain exists is designing on sand. +3. Note the module's status in §2.4 (ported / in-flight / untouched). + +## Step 2 — Measure the shape + +```bash +scripts/routine-map.sh +``` + +One row per procedure, sorted by **MAXLOOP** — the longest single loop body. That is the metric +that matters. A 600-line routine of small loops ports fine; a 600-line routine whose *one* loop +body is 450 lines is the monolith problem, and `LINES` alone cannot tell them apart. + +Calibration from this tree (`references/calibration.md` has the full table and how to regenerate): + +| Routine | LINES | MAXLOOP | Verdict | +|---|---|---|---| +| `find_uv_at_h` | 123 | 65 | **the blessed template** — ported as-is | +| `set_diffusivity` | 624 | 269 | refactor first | +| `KPP_compute_BLD` | 522 | 369 | research-grade | +| `applyBoundaryFluxesInOut` | 664 | 452 | refactor first | +| `ePBL_column` | 1061 | 701 | naive port only (see below) | + +## Step 3 — Map the callees (the triage input) + +```bash +scripts/callees.sh +``` + +**`MAXDEP` is the design-critical column.** A callee at `MAXDEP 0` is host orchestration outside +the loops — free, irrelevant to the port. A callee at `MAXDEP > 0` is *inside the loop nest*: to +port that loop, it must become device-callable, or be hoisted out first. + +So: **the loop-interior call list is the triage.** The blessed template has an empty one — + +``` +find_uv_at_h: every callee MAXDEP 0 -> portable as-is +applyBoundaryFluxesInOut: mom_error MAXDEP 4, forcing_SinglePointPrint MAXDEP 4, + post_data MAXDEP 1, 2 generic interfaces MAXDEP 2 -> refactor first +``` + +`callees.sh` only catches `call` statements — Fortran function references are indistinguishable +from array indexing without a symbol table. **Read the loop bodies for function references** +(EOS elementals, `ratio_max`, `cuberoot`); they carry the same constraint. + +## Step 4 — The refactor triage + +Classify every callee with `MAXDEP > 0`: + +| Class | Consequence | Action before porting | +|---|---|---| +| **HOST-SINK** (`MOM_error`, `post_data`, chksum, halo, I/O) | **Blocks the port outright** — never-do #10: no allocate, I/O or `post_data` in a device loop | Hoist out, or return status via `intent(out)` flags and fold host-side after (the `efp_decompose` model) | +| **GENERIC-INTERFACE** / `class(...)` dispatch | Device-fatal or silently wrong (doc 06; never-do #2) | Port the `_loc` free function first (doc 06 §6.3) | +| **External pkg** (`cvmix_*`, FMS) | Largest `declare target` surface; may be unportable | Size it *before* committing to the port | +| **In-file, not `pure`** | Silent wrong numbers if it fails to inline (`3cb184edd`) | Make `pure` + `declare target`, or `FORCEINLINE` | +| **In-file, already `pure`** | Cheap | `declare target` | +| *(list is empty)* | — | **Port in place** — go to `knowledge/KNOWLEDGE.md` §3 | + +**Refactor first if any of:** a HOST-SINK is called inside the loop nest; the loop body writes +module state (blocks `pure`); polymorphic dispatch happens inside the loop; or MAXLOOP is large +*and* the body mixes ≥2 distinct concerns. Otherwise port in place. + +Treat the thresholds as calibration, not law. The judgement is **"can one reviewer hold this loop +body and its bitwise argument in their head at once?"** — 65 lines with no interior calls, yes; +452 lines with `MOM_error` at depth 4, no. + +The counter-example that keeps this honest: **`ePBL_column`** (1061/701) has 8 loop-interior +callees but *zero* host sinks and *zero* polymorphic dispatch — all in-file, all extractable. That +self-contained call graph is why `origin/epbl-3d`'s naive whole-column `pure` + `do concurrent` +worked at all ("100x… ~2ms/step"). Shape alone would have condemned it. It still needed manual +inlining of `get_Langmuir_Number` and `find_mstar` — **the exact two callees `callees.sh` flags at +MAXDEP 2** — and it has no CPU-preserving story, so it fails ground rule 2. Big MAXLOOP means +*"read the call list before judging"*, not *"refuse"*. + +## Step 5 — The only legal refactor: verbatim extraction + +Ground rule 1 forbids reordering floating-point arithmetic. So "refactor to make porting easier" +has exactly **one** blessed form (doc 08 §4): **extract the innermost side-effect-free arithmetic +into a `pure`/`elemental` procedure, moving code, never reordering it.** + +1. Find the innermost side-effect-free arithmetic span inside the hot loop. +2. Move it **verbatim** into a new `pure`/`elemental` procedure. Copy expressions + character-for-character. Do not "tidy" parentheses — Fortran parens pin evaluation order. +3. Route side effects out as `intent(out)` flags; the host caller folds them into module state + *after* the loop. This is exactly why `efp_decompose` is `pure`: it reports `is_nan`/`is_ovf` + rather than touching the module error flags (`MOM_coms.F90:779`). It is also the answer to a + `MOM_error(FATAL)` at depth 4. +4. If callers need the old API, keep the original as a **thin wrapper** delegating to the new + procedure — the EOS `_loc` model (`52a1b3954`). +5. Add `!$omp declare target` after all declarations, before the first executable. +6. **Gate the refactor on CPU alone, bitwise, as its own commit — before any directive exists.** + +Step 6 is the whole argument for refactoring first. A verbatim extraction **must** be bit-identical +on CPU; if it isn't, the extraction wasn't verbatim, and you learn that from a cheap CPU run +instead of from a GPU checksum mismatch tangled up with mapping bugs. It splits a scary 664-line +port into two separately-auditable diffs: + +``` +commit 1: verbatim extraction, zero directives -> gate: CPU bitwise identical +commit 2: the port (KNOWLEDGE.md §3) -> gate: §3 Step 9 (GPU bitwise + ≥2 GPUs) +``` + +**Not refactors — these change bits and are forbidden** (§7.2): re-associating arithmetic; +distributing parentheses; splitting a producer loop from the reduction that consumes it +(`5f413739b` — even ifort re-associates, even at `-O0`); fusing loops that don't meet doc 05 §4's +conditions. And note declaration order of large stack arrays measurably moves CPU performance +(`28eb296f4`, "Move with caution!") — reordering declarations is not free either. + +## Step 6 — Hazard audit + +Run `knowledge/KNOWLEDGE.md` §3 Step 2 against the module — do not re-derive it here. It covers pointer +members and `associated()` flow, restart-registered fields, EOS forms exercised, recurrences, halo +calls, diagnostics, and float reductions. Two that most often change the *design* rather than the +code: + +- **Recurrences** (`x(k)` depends on `x(k±1)`): these pick the loop form for you (§4.1 branch 5, + teams-loop + serial k) and disqualify k-blocking outright (doc 05 §7.0). +- **EOS forms**: only buggy-Wright and Roquet_rho have GPU-safe direct kernels. The **default + `WRIGHT_FULL` is still polymorphic and device-fatal**. If your target needs another form on + device, that form is a prerequisite port, not a detail. + +## Step 7 — Loop-form plan + +For each loop, take the first matching branch of the `knowledge/KNOWLEDGE.md` §4.1 decision tree and record +the choice plus its reason. Do not invent forms. Note where k-blocking applies (§3 Step 3) and +whether the module needs its own `nkblock` CS parameter. + +## Step 8 — Emit the design + +Lead with the verdict — **port in place** or **refactor first** — and the one fact that decided it. +Then: + +1. **Shape** — LINES/MAXLOOP table for the target routines. +2. **Call map** — loop-interior callees only, each classified per Step 4, each with its action. +3. **Verdict + reasoning**, naming the blocking callee or hazard if refactoring. +4. **Refactor plan** (if any) — what gets extracted verbatim, what flags replace what side effects, + what the CPU gate is. +5. **Port plan** — loop-form per loop; k-blocking yes/no; data-mapping handed to + `gpu-data-residency`; halo strategy (§3 Step 7). +6. **Prerequisites** — EOS forms, in-flight branches to merge first, dependency-order items. +7. **Sequenced commits, each with its gate.** +8. **Open questions** — things needing a run, a profile, or a maintainer decision. Say so rather + than guessing; §8 and §9 show which questions source-reading genuinely cannot close. + +## Anti-patterns + +- **Designing against `LINES`.** MAXLOOP and the loop-interior call list decide; total length does not. +- **"Refactor" that isn't verbatim.** If you retyped an expression, you changed it. Copy it. +- **Skipping §2.4.** Restarting work an in-flight branch already did is the most expensive mistake available. +- **Porting a routine whose EOS form isn't ported.** The prerequisite is the work. +- **Naive-vs-blessed by default.** `epbl-3d` is fast and GPU-only; ground rule 2 requires one source + form serving both targets. Choose explicitly and say which you chose (doc 10's 8-gate rubric). +- **Treating a big MAXLOOP as a refusal.** Read the call list first — `ePBL_column` is the lesson. diff --git a/skills/mom6-gpu-architect/references/calibration.md b/skills/mom6-gpu-architect/references/calibration.md new file mode 100644 index 0000000..e626e00 --- /dev/null +++ b/skills/mom6-gpu-architect/references/calibration.md @@ -0,0 +1,84 @@ +# Calibration: what a portable routine looks like in this tree + +Measured on `dev/gpu` HEAD, 2026-07-14. **Regenerate rather than trust** — branches land: + +```bash +scripts/routine-map.sh +scripts/callees.sh +``` + +## The shape spectrum + +| Routine | File | LINES | MAXLOOP | Loop-interior callees | Status | +|---|---|---|---|---|---| +| `find_uv_at_h` | `MOM_diabatic_aux.F90:495` | 123 | 65 | **none** (all MAXDEP 0) | **ported — the template** | +| `set_diffusivity` | `MOM_set_diffusivity.F90:244` | 624 | 269 | 13+, incl. 27× `hchksum` @1, `thickness_to_dz` generic @2 | untouched, Tier-1 #2 | +| `KPP_compute_BLD` | `MOM_CVMix_KPP.F90:996` | 522 | 369 | 6, incl. 3× external `cvmix_kpp_*` @2–3, `calculate_density` generic @2 | untouched, Tier-1 #4 | +| `applyBoundaryFluxesInOut` | `MOM_diabatic_aux.F90:685` | 664 | 452 | 5, incl. `MOM_error` @4, `forcing_SinglePointPrint` @4, `post_data` @1, 2 generics @2 | untouched | +| `ePBL_column` | `MOM_energetic_PBL.F90:896` | 1061 | 701 | 8, **all in-file, no host sinks, no polymorphism** | naive port on `origin/epbl-3d` | + +The two ends of the spectrum are the whole lesson: + +- **`find_uv_at_h`** — 123 lines, MAXLOOP 65, and its *entire* call list (`cpu_clock_begin`, + `MOM_error`, `cpu_clock_end`) sits at MAXDEP 0, outside the loops. The loop bodies are + self-contained arithmetic. That is why it ported cleanly as `target teams loop collapse(2)` + + serial-k, and why `knowledge/KNOWLEDGE.md` §6 Tier-1 #1 says to copy it verbatim for the diabatic + tridiagonals. +- **`applyBoundaryFluxesInOut`** — 664 lines, one `do j` at `:871` spanning to `:1324` (452-line + body), with `MOM_error(FATAL)` and `forcing_SinglePointPrint` **four loops deep** at `:1167`/`:1174`. + Those are never-do #10 outright. No amount of directive placement fixes it; the arithmetic has to + come out into a `pure` column kernel with error flags first. + +## Why the call list beats the shape metric + +`ePBL_column` is the counter-example that stops MAXLOOP from becoming a dumb threshold. It is the +**biggest** loop body in the Tier-1 set (701) — and it is the one with a *clean* interior call +list: 8 callees, all in-file or a single cross-module helper, no host sinks, no generic interfaces. +That is exactly why `origin/epbl-3d`'s naive whole-column `pure` + `do concurrent(j,i)` worked +("100x… ~2ms/step"). + +`callees.sh` flags its interior callees at MAXDEP 2 as `get_langmuir_number` (cross-module, +`MOM_wave_interface.F90`) and `find_mstar` (in-file `:3522`). `knowledge/KNOWLEDGE.md` §6 Tier-1 #3 +independently records that epbl-3d "needed two workarounds to reuse deliberately: **manual inlining +of `get_Langmuir_Number`/`find_mstar`**, and fixed-size column arrays". The tool predicts the +documented workarounds from the call graph alone — which is the evidence that the loop-interior +call list is the right triage input. + +Same check on `KPP_compute_BLD`: the tool flags 3 external `cvmix_kpp_*` calls at depth 2–3; +§6 Tier-1 #4 independently says its hazard is that it "calls into external `pkg/CVMix-src` — the +largest `declare target`/inlining surface of Tier-1". Agreement, derived two different ways. + +So: **big MAXLOOP means "read the call list before judging", not "refuse".** But note that +epbl-3d's success is only half a result — it is GPU-only, with no CPU-preserving story, so it +fails ground rule 2. A clean call graph tells you the port is *mechanically* possible; ground +rule 2 still decides whether that port is *acceptable*. + +## Reading `routine-map.sh` output + +``` + LINES MAXLOOP DEPTH CALLS PURE START NAME + 664 452 3 23 - 685 applyboundaryfluxesinout + 123 65 3 3 - 495 find_uv_at_h +``` + +- **MAXLOOP / LINES ratio** is the monolith tell: 452/664 = 68% of the routine inside one loop. +- **DEPTH** 3–4 with a large MAXLOOP means the extraction target is probably the *innermost* + full-column body, not the outer `do j`. +- **PURE** already `pure`/`elemental` → cheap to make device-callable; just needs `declare target`. + Most of `MOM_EOS_Wright.F90`'s elementals show `pure` with MAXLOOP 0 — the shape you are + refactoring *toward*. +- Both scripts strip comments and string literals before counting, so `MOM_error("... do ...")` + cannot fake a loop. They are heuristic reading aids, not parsers — verify the boundaries of any + loop you are about to act on (`sed -n '871p;1324p' `). + +## Known blind spots + +1. **Function references are invisible to `callees.sh`.** Only `call` statements are caught. EOS + elementals, `ratio_max`, `cuberoot` are function calls and carry the identical device-callable + constraint. Read the loop bodies. +2. **`? (external/pkg, or generic interface)`** means the definition isn't under `src/` — usually + `pkg/` (CVMix) or FMS. Not benign: it is the largest inlining surface there is. +3. **`GENERIC-INTERFACE`** means the call dispatches through a generic. On device that is the + doc 06 polymorphic hazard — resolve which specific it binds to before designing around it. +4. Neither script understands `#ifdef`. A routine's shape can differ between the CPU and + `__NVCOMPILER_OPENMP_GPU` builds. diff --git a/skills/mom6-gpu-architect/scripts/callees.sh b/skills/mom6-gpu-architect/scripts/callees.sh new file mode 100755 index 0000000..af19b26 --- /dev/null +++ b/skills/mom6-gpu-architect/scripts/callees.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# callees.sh +# +# Every `call` made by one routine, with the LOOP DEPTH at the call site and where the callee is +# defined. Depth is the design-critical column: +# +# DEPTH 0 host orchestration, outside any loop -- free, needs no device work +# DEPTH >0 inside the loop nest -- to port that loop, this callee MUST become device-callable +# (pure/elemental + declare target, or force-inlined), or be hoisted out first. +# +# WHERE resolves the definition: in-file (with pure/elemental noted), or the module that defines +# it, or HOST-SINK for known host-only consumers. +# +# LIMITATION: only `call` statements. Fortran function references are indistinguishable from array +# indexing without a symbol table, so device-relevant *functions* (EOS elementals, ratio_max, ...) +# will NOT appear here -- read the loop bodies for those. + +set -euo pipefail +[ "$#" -ge 2 ] || { echo "usage: $(basename "$0") " >&2; exit 2; } +file=$1; want=$2 +[ -r "$file" ] || { echo "cannot read: $file" >&2; exit 2; } + +# Resolve callee definitions against the repo's src/ tree, not the caller's cwd -- otherwise every +# WHERE silently degrades to "?", which reads as "external" and is a false signal. +ROOT=$(cd "$(dirname "$file")" && git rev-parse --show-toplevel 2>/dev/null || true) +if [ -z "${ROOT:-}" ] || [ ! -d "$ROOT/src" ]; then ROOT=.; fi +[ -d "$ROOT/src" ] || echo "warning: no src/ tree found from $file -- WHERE will be unresolved" >&2 + +HOST_SINKS='post_data|pass_var|pass_vector|start_group_pass|complete_group_pass|hchksum|uchksum|vchksum|uvchksum|bchksum|chksum|save_restart|register_restart_field|register_diag_field|max_across_pes|min_across_pes|sum_across_pes|global_area_mean|global_area_integral|mom_error|forcing_singlepointprint|get_param|cpu_clock_begin|cpu_clock_end|calltree_enter|calltree_leave|calltree_waypoint' + +awk -v want="$(printf '%s' "$2" | tr 'A-Z' 'a-z')" -v sinks="$HOST_SINKS" ' +{ + raw = $0; line = tolower($0) + sub(/!.*/, "", line); gsub(/"[^"]*"/, "", line); gsub(/'"'"'[^'"'"']*'"'"'/, "", line) + + if (line ~ /^[ \t]*end[ \t]*(subroutine|function)/) { if (inr) exit; next } + if (line !~ /^[ \t]*end[ \t]/ && line ~ ("(^|[ \t])(subroutine|function)[ \t]+" want "([ \t]*\\(|[ \t]*$)")) { + inr = 1; sp = 0; next + } + if (!inr) next + + nclose = gsub(/(^|[;[:space:]])end[[:space:]]*do([;[:space:]]|$)/, " ", line) + nopen_line = line + nopen = gsub(/(^|[;[:space:]])do([[:space:]]|$)/, " ", line) + for (n = 0; n < nclose; n++) if (sp > 0) sp-- + + tmp = line + while (match(tmp, /(^|[;[:space:]])call[[:space:]]+[a-z_][a-z_0-9]*/)) { + s = substr(tmp, RSTART, RLENGTH); sub(/^.*call[[:space:]]+/, "", s) + key = s + if (!(key in cnt)) { order[++nord] = key; firstline[key] = NR; maxd[key] = sp } + if (sp > maxd[key]) maxd[key] = sp # deepest call site is what constrains the port + cnt[key]++ + tmp = substr(tmp, RSTART + RLENGTH) + } + for (n = 0; n < nopen; n++) sp++ +} +END { + printf "%-40s %5s %7s %s\n", "CALLEE", "N", "MAXDEP", "FIRST" + for (i = 1; i <= nord; i++) { + k = order[i] + printf "%-40s %5d %7d %d\n", k, cnt[k], maxd[k], firstline[k] + } +} +' "$file" | while IFS= read -r row; do + name=$(printf '%s' "$row" | awk '{print $1}') + case "$name" in + CALLEE) printf '%s %s\n' "$row" "WHERE"; continue ;; + esac + low=$(printf '%s' "$name" | tr 'A-Z' 'a-z') + if printf '%s' "$low" | grep -qE "^($HOST_SINKS)$"; then + where="HOST-SINK" + elif def=$(grep -niE "^ *([a-z_ ()]* )?(subroutine|function) +$name *\(" "$file" | head -1); then + if printf '%s' "$def" | grep -qiE "(pure|elemental)"; then where="in-file:$(printf '%s' "$def" | cut -d: -f1) (pure)" + else where="in-file:$(printf '%s' "$def" | cut -d: -f1)"; fi + else + hit=$(grep -rliE "^ *([a-z_ ()]* )?(subroutine|function) +$name *\(" "$ROOT/src" 2>/dev/null | head -1 || true) + if [ -n "$hit" ]; then + where=${hit#$ROOT/} + elif grep -rqiE "^ *interface +$name *$" "$ROOT/src" 2>/dev/null; then + # No concrete definition, but a generic interface exists: the call dispatches through it. + # On the device that is the doc 06 polymorphic hazard -- resolve which specific it binds to. + where="GENERIC-INTERFACE (polymorphic hazard, doc 06)" + else + where="? (external/pkg, or generic interface)" + fi + fi + printf '%s %s\n' "$row" "$where" +done diff --git a/skills/mom6-gpu-architect/scripts/routine-map.sh b/skills/mom6-gpu-architect/scripts/routine-map.sh new file mode 100755 index 0000000..2a888ec --- /dev/null +++ b/skills/mom6-gpu-architect/scripts/routine-map.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# routine-map.sh +# +# Triage table: one row per procedure, sorted by MAXLOOP (the longest single loop body). +# MAXLOOP is the shape metric that matters -- a routine that is 600 lines of small loops ports +# fine; a routine with one 200-line loop body is the "1500 lines of a single do loop" problem +# and wants a verbatim extraction before any directive goes near it. See SKILL.md Step 3. +# +# LINES total lines in the procedure +# MAXLOOP longest loop body (lines between a do and its matching enddo) +# DEPTH deepest loop nesting +# CALLS number of `call` statements +# PURE procedure is already pure/elemental (cheap to make device-callable) +# +# Heuristic reading aid, not a parser. Fortran is case-insensitive; comments and string +# literals are stripped before counting so `MOM_error("... do ...")` cannot fake a loop. + +set -euo pipefail +[ "$#" -ge 1 ] || { echo "usage: $(basename "$0") " >&2; exit 2; } +[ -r "$1" ] || { echo "cannot read: $1" >&2; exit 2; } + +awk ' +function flush_routine() { + if (name != "") printf "%6d %8d %6d %6d %-4s %6d %s\n", NR-start, maxloop, maxdepth, ncalls, (ispure?"pure":"-"), start, name + name = ""; maxloop = 0; maxdepth = 0; ncalls = 0; ispure = 0; sp = 0 +} +{ + line = tolower($0) + sub(/!.*/, "", line) # strip trailing comment + gsub(/"[^"]*"/, "", line) # strip string literals + gsub(/'"'"'[^'"'"']*'"'"'/, "", line) + + if (line ~ /^[ \t]*end[ \t]*(subroutine|function)/) { flush_routine(); next } + + if (line !~ /^[ \t]*end[ \t]/ && line ~ /(^|[ \t])(subroutine|function)[ \t]+[a-z_][a-z_0-9]*/ \ + && line !~ /(^|[ \t])(module|abstract|procedure)[ \t]/) { + if (name != "") flush_routine() + match(line, /(subroutine|function)[ \t]+[a-z_][a-z_0-9]*/) + nm = substr(line, RSTART, RLENGTH); sub(/^(subroutine|function)[ \t]+/, "", nm) + name = nm; start = NR; maxloop = 0; maxdepth = 0; ncalls = 0; sp = 0 + ispure = (line ~ /(^|[ \t])(pure|elemental)[ \t]/) + next + } + if (name == "") next + + ncalls += gsub(/(^|[;[:space:]])call[[:space:]]+[a-z_]/, " ", line) + + nclose = gsub(/(^|[;[:space:]])end[[:space:]]*do([;[:space:]]|$)/, " ", line) + nopen = gsub(/(^|[;[:space:]])do([[:space:]]|$)/, " ", line) + + for (n = 0; n < nopen; n++) { stack[sp++] = NR; if (sp > maxdepth) maxdepth = sp } + for (n = 0; n < nclose; n++) { + if (sp > 0) { body = NR - stack[--sp] - 1; if (body > maxloop) maxloop = body } + } +} +END { flush_routine() } +' "$1" | sort -k2,2rn | awk 'BEGIN{printf "%6s %8s %6s %6s %-4s %6s %s\n","LINES","MAXLOOP","DEPTH","CALLS","PURE","START","NAME"} {print}' diff --git a/skills/mom6-gpu-programmer/SKILL.md b/skills/mom6-gpu-programmer/SKILL.md new file mode 100644 index 0000000..575308c --- /dev/null +++ b/skills/mom6-gpu-programmer/SKILL.md @@ -0,0 +1,160 @@ +--- +name: mom6-gpu-programmer +description: End-to-end driver for MOM6 GPU porting work. Takes a routine or module from "should we port this?" through design, bitwise-safe refactor, port, verification and merge review — or evaluates an existing port, branch or diff. Use when asked to "port ", "start a port", "what would it take to port X", "evaluate/audit this port", "review this GPU diff", "is this port ready to merge", or when debugging wrong GPU answers. Orchestrates the mom6-gpu-architect and gpu-data-residency skills and owns the gates between them. +--- + +# MOM6 GPU programmer + +The driver for a whole piece of GPU porting work. It **sequences** the other skills and **owns the +gates between them**; it does not restate what they say. + +| Need | Where it lives | +|---|---| +| shape, call map, refactor-or-port triage | the **`mom6-gpu-architect`** skill | +| where to map arrays, where to copy back | the **`gpu-data-residency`** skill | +| porting mechanics (k-blocking, loop forms, halos, verification) | `knowledge/KNOWLEDGE.md` §3, §4 | +| symptom → fix | `knowledge/KNOWLEDGE.md` §5 | +| merge review rubric (8 gates) | `knowledge/gpu-knowledge/10-inflight-ports.md:455` | + +Invoke a child skill with the **Skill tool** at the phase that needs it (`mom6-gpu-architect` at +Phase 1, `gpu-data-residency` at Phase 3). They are separate skills, so calling them from here is +fine: the Skill tool's only stated prohibition is re-invoking a skill that is *already running* — +parent→child yes, self-recursion no. + +Two practical notes: + +- **Invoke at the phase that needs it, not all up front.** A child's full SKILL.md enters the + conversation on invocation and stays there for the session. Phase 1 and Phase 3 are far apart; + loading both at the start just burns context. +- If a child's content is already in this conversation, **use it directly — do not re-invoke**. + Nested skill-to-skill invocation is not spelled out in the public Skills documentation, so if it + ever misbehaves the fallback is identical in effect: `Read skills//SKILL.md`. + +The user can also chain these directly from the command line (`/mom6-gpu-architect /gpu-data-residency +`) — this skill is for when the *sequencing and the gates between* the phases are the point. + +## Prime directive: never claim a gate you did not run + +Ground rule 6 (`KNOWLEDGE.md:60`): **study source + git only unless explicitly told to build.** +The acceptance gate (§3 Step 9) requires a build *and* a ≥2-GPU run. + +Those two facts collide, and the collision is the most important thing about this skill: + +> **By default you can produce a port. You cannot accept one.** + +Check the boundary before promising anything: `which nvfortran mpirun`. On a typical session here +they are absent — Phase 4 is then a **hand-back**, not a step you perform. + +So: never write "verified", "bitwise-identical", "confirmed", or "passes" about a run you did not +execute. Write **"unverified — here is the exact experiment"** and name the config, the ranks, and +the fields to checksum. `knowledge/KNOWLEDGE.md` §9's open list is open precisely because source-reading +cannot close those questions; quietly adding false certainty to that pile is the worst available +contribution. §8's own header models the standard: *"source-and-git … no builds or runs."* + +## Mode: PORT + +### Phase 0 — Orient ⟨STOP-GATE: is this even the right target?⟩ + +1. Read `knowledge/KNOWLEDGE.md` (§2 architecture, §6 work queue) if not already in context. +2. **In-flight check** (§2.4): is there a branch carrying groundwork? If yes → **STOP and report**. + Restarting work a branch already did is the most expensive mistake available. +3. **Dependency check** (§6): Tier 1 is ordered — EOS `_loc` coverage → N²/density inputs → + `set_diffusivity` → KPP/EPBL → `kappa_shear`. If a prerequisite is unported → **STOP and report + the chain**, don't design on sand. +4. **EOS check**: does the target need a form on device other than buggy-Wright / Roquet_rho? The + default `WRIGHT_FULL` is still polymorphic and device-fatal. That form is then the real work item. + +### Phase 1 — Design ⟨STOP-GATE: verdict recorded⟩ + +Invoke **`mom6-gpu-architect`**. It returns: shape (LINES/MAXLOOP), the loop-interior call list, +and a **port-in-place vs refactor-first** verdict. + +If the verdict is refactor-first and the extraction is substantial → **STOP and confirm scope with +the user** before writing code. A refactor is a separate piece of work with its own risk. + +### Phase 2 — Refactor (only if triaged) ⟨GATE: CPU bitwise identical⟩ + +The only legal refactor is **verbatim extraction** into `pure`/`elemental` (doc 08 §4) — move code, +never reorder it. Recipe in the architect skill, Step 5. + +**This is its own commit, with zero directives in it, gated on CPU alone.** A verbatim extraction +*must* be bit-identical on CPU; if it isn't, the extraction wasn't verbatim — and you learn that +from a cheap CPU run instead of from a GPU checksum mismatch tangled up with mapping bugs. + +If you cannot run the CPU gate → the extraction diff is still deliverable, but it is **unverified**. +Say so and stop; do not stack the port on top of an unverified refactor. + +### Phase 3 — Port + +Follow `knowledge/KNOWLEDGE.md` §3 Steps 3–8 for the mechanics. For **every data-mapping decision** — where +`enter data` goes, `to` vs `alloc`, which host consumer forces an `update from` — invoke +**`gpu-data-residency`** rather than improvising. Mapping bugs are the silent class. + +Keep the §7.2 never-do list open while writing. The ones that bite during a port: +no early `exit`/`return`/`cycle` in device loops; no shared-scalar writes without `reduce`; +no `map(delete:)` on an object this scope doesn't own; every `enter data` gets a mirrored `exit data`. + +### Phase 4 — Verify ⟨GATE: §3 Step 9 — usually a hand-back⟩ + +The gate is: bit-identical `MOM_checksums` field checksums + EFP `write_energy`, block-size +invariance (`nkblock` 0/1/nz agree), and **≥2 ranks/GPUs** — single-GPU correctness does not prove +a port; the `alloc`-vs-`to` and missing-`reduce` bugs are latent until multi-device. + +If you cannot build: produce the diff plus **the experiment** — exact config, rank count, fields to +checksum, and what a failure would mean. Then **STOP and hand back**. This is a legitimate, +expected outcome, not a failure. + +### Phase 5 — Merge review ⟨GATE: all 8⟩ + +Run doc 10's 8-gate rubric (`10-inflight-ports.md:455-508`) against your own diff *verbatim* — +don't paraphrase it, it cites the evidence for each gate. Summary of what it checks: block-size CS +params `#ifdef`-gated; one persistent data region per hot path; `do concurrent` as the default +idiom; EOS via the 2D/3D `_loc` interface with the v-table resolved host-side; an explicit bitwise +argument naming which runtime path you hit; no debug prints; CPU defaults actually benchmarked (not +copy-pasted); k-recurrences nested `do concurrent(j) → serial do k → do concurrent(i)`. + +Gate 7 (CPU tuning benchmarked) is another one you probably cannot close in-session. Say so. + +## Mode: EVALUATE + +For "audit this port", "is this ready", "review this diff", or "why are the answers wrong". + +1. **Scope it** — a diff, a branch, a module, or a symptom. For a branch: `git diff dev-gfdl...`. +2. **If a symptom is reported**, go to `knowledge/KNOWLEDGE.md` §5's symptom index *first*, and check **row 0 + first, always**: your own diff (missing map, misplaced accumulation, unbalanced enter/exit) + before blaming nvfortran. Rows marked `SILENT` are the dangerous class. +3. **Map/transfer audit** → invoke **`gpu-data-residency`** (its Step 5 balance checks and Step 6 + report format). +4. **Shape/call audit** (if the question is "how hard is this?") → invoke **`mom6-gpu-architect`**. +5. **Merge readiness** → doc 10's 8 gates. +6. **Report** with findings separated into **confirmed** (both sites read, branch structure checked), + **suspected** (needs a run), and **needs a maintainer decision**. Give each finding: the two + sites, the predicate that reaches the bad path, the symptom it produces, and the one-line fix. + +Known latent items worth checking against before reporting something as new: §8, "six early-`exit`-under-`do concurrent` sites remain at HEAD" (six +early-`exit`-under-DC sites still at HEAD), finding B (the `ADp` mapping lifecycle), and +`gpu-data-residency`'s `khdt_x` worked example. + +## Stop and ask — do not push through + +- An in-flight branch already covers the target (§2.4). +- A prerequisite is unported (EOS form, density inputs). +- The triage says refactor-first and the extraction is large. +- **Naive vs blessed**: ground rule 2 requires one source form serving CPU and GPU. `epbl-3d` is + fast and GPU-only. That trade is a maintainer decision — surface it, don't make it. +- A gate needs a build/run you cannot do. +- **You find a bug in existing code.** Report it with its reaching predicate; do not silently fix it + inside an unrelated port — it belongs in its own diff with its own gate. +- The question is genuinely unclosable from source (the §9 open-question shape). + +Don't commit or push unless asked. When you do, the sequence is: refactor commit (CPU gate) → +port commit (GPU gate) — separately auditable, never squashed together. + +## Never + +1. Claim a verification you didn't run, or let "compiles" stand in for "bitwise". +2. Accept a nonzero checksum diff as rounding. One differing bit means the port is wrong. +3. Reorder floating-point arithmetic — including "tidying" parentheses (§7.2 #1). +4. Port past a `STOP-GATE` because the next phase looks easy. +5. Restate the child skills' content here instead of invoking them. +6. Write outside the repo; temp artifacts → `tmp_local_artifacts/`.