MOM6-GPU · JorgeG94 · Jul 15, 2026 · Jul 16, 2026
diff --git a/knowledge/KNOWLEDGE.md b/knowledge/KNOWLEDGE.md
diff --git a/knowledge/README.md b/knowledge/README.md
@@ -0,0 +1,40 @@
+# MOM6 GPU knowledge base
+
+The knowledge the `skills/` in this repo read from. `KNOWLEDGE.md` is the operational playbook
+(porting procedure, decision rules, symptom→fix index, prioritized work queue, proven-works and
+never-do lists); `gpu-knowledge/00-14` are the deep-dive docs behind it, plus standalone bug notes.
+
+Read `KNOWLEDGE.md` first — it is self-contained, and every load-bearing rule is inlined with its
+citation so you can act without opening the deep docs.
+
+## Provenance — read this before trusting a line number
+
+The knowledge base cites MOM6 source by `file:line`, plus commit hashes and branch names. Those are
+only meaningful against a specific tree:
+
+| | |
+|---|---|
+| `file:line` refs valid against | `dev/gpu` @ **`c82e1254a`** |
+| upstream baseline | `dev-gfdl` @ `c3237e27f` |
+| built | 2026-07-14, from source + git only — no builds or runs |
+
+**Line numbers rot**, and they rot invisibly here: there is no MOM6 source in this repo to
+contradict a stale reference. Verify any anchor before acting on it. Commit hashes and branch names
+do not rot — prefer them where both are available.
+
+## The confidence markers are load-bearing
+
+`KNOWLEDGE.md` §8 and §9 record, per claim, which were **verified from source**, which are
+**advanced but need a run or a profile to close**, and which are **open maintainer decisions**. The
+same distinction appears in the bug notes. That grading is the most valuable thing in here — please
+preserve it when editing rather than flattening everything to assertion.
+
+Several findings are explicitly **unconfirmed** and say so. They are recorded because they are
+worth checking, not because they are established.
+
+## Status
+
+Current best practice, not a finished spec. The port is a work in progress and only the code paths
+exercised by the `benchmark` / `benchmark_ALE` configurations have been validated. When you hit a
+pattern these docs do not cover, make a decision consistent with the philosophy, flag it, and add it
+here once it is confirmed.
diff --git a/knowledge/gpu-knowledge/00-architecture.md b/knowledge/gpu-knowledge/00-architecture.md
diff --git a/knowledge/gpu-knowledge/01-memory-control-structures.md b/knowledge/gpu-knowledge/01-memory-control-structures.md
diff --git a/knowledge/gpu-knowledge/02-pointer-usage.md b/knowledge/gpu-knowledge/02-pointer-usage.md
diff --git a/knowledge/gpu-knowledge/03-openmp-mapping.md b/knowledge/gpu-knowledge/03-openmp-mapping.md
diff --git a/knowledge/gpu-knowledge/04-do-concurrent-patterns.md b/knowledge/gpu-knowledge/04-do-concurrent-patterns.md
diff --git a/knowledge/gpu-knowledge/05-kblocking-tiling.md b/knowledge/gpu-knowledge/05-kblocking-tiling.md
diff --git a/knowledge/gpu-knowledge/06-eos-layer.md b/knowledge/gpu-knowledge/06-eos-layer.md
diff --git a/knowledge/gpu-knowledge/07-reproducibility.md b/knowledge/gpu-knowledge/07-reproducibility.md
diff --git a/knowledge/gpu-knowledge/08-cross-module-inlining.md b/knowledge/gpu-knowledge/08-cross-module-inlining.md
diff --git a/knowledge/gpu-knowledge/09-barotropic-solver.md b/knowledge/gpu-knowledge/09-barotropic-solver.md
diff --git a/knowledge/gpu-knowledge/10-inflight-ports.md b/knowledge/gpu-knowledge/10-inflight-ports.md
diff --git a/knowledge/gpu-knowledge/11-halos-domains-multigpu.md b/knowledge/gpu-knowledge/11-halos-domains-multigpu.md
diff --git a/knowledge/gpu-knowledge/12-diagnostics-io.md b/knowledge/gpu-knowledge/12-diagnostics-io.md
diff --git a/knowledge/gpu-knowledge/13-compiler-workarounds.md b/knowledge/gpu-knowledge/13-compiler-workarounds.md
diff --git a/knowledge/gpu-knowledge/14-vertical-physics-ale-status.md b/knowledge/gpu-knowledge/14-vertical-physics-ale-status.md
diff --git a/knowledge/gpu-knowledge/nvfortran-automatic-array-bug.md b/knowledge/gpu-knowledge/nvfortran-automatic-array-bug.md
@@ -0,0 +1,85 @@
+# nvfortran: automatic arrays in device-called procedures
+
+**One line:** nvfortran cannot allocate an **automatic array on the device stack** when its size
+comes from a runtime expression that is **not a dummy argument**, inside a procedure **called from**
+a device region.
+
+## Symptom
+
+Device compile failure (or a forced fixed-size workaround) on a local array like:
+
+```fortran
+real, dimension(SZK_(GV)+1) :: dz_col     ! SZK_(GV) -> GV%ke : a derived-type component
+```
+
+in a routine invoked from inside a `!$omp target`/`do concurrent` region. The compiler needs a
+per-thread stack size it cannot know: `GV%ke` is a runtime value reached through a derived type,
+and there is no device-side dynamic stack to fall back on.
+
+## What it is *not*
+
+It is **not** "automatic arrays can't be privatized", and it is **not** specific to
+`do concurrent`. This is merged on `dev/gpu` and bitwise-gated, and it privatizes an automatic
+array just fine:
+
+```fortran
+subroutine vertvisc                        ! MOM_vert_friction.F90
+  real :: c1(SZK_(GV))                     ! :575  automatic, sized GV%ke
+
+  !$omp target teams loop collapse(2) &    ! :737  WORKS
+  !$omp   private(b1, c1, d1, Ray, b_denom_1)
+```
+
+## The actual distinction
+
+| Shape | Works? | Why |
+|---|---|---|
+| Automatic array in the directive's **own routine**, listed in `private(...)` | **Yes** | the host sizes it at routine entry; the compiler emits N per-thread copies |
+| Automatic array **local to a callee** invoked from a device region, sized from non-dummy data | **No** | would need device-side dynamic stack allocation |
+
+## Evidence
+
+`05c74b56b` — Marshall Ward, 2026-06-03, *"ePBL: Replace auto array size (nk=75)"*, on the
+epbl-3d branch (`MOM_energetic_PBL_smod.F90`):
+
+```fortran
+!  in ePBL_column_3d -- CALLED from the device loop at :528
+-  real, dimension(SZK_(GV)+1) :: ...
++  real, dimension(75+1)       :: ...      ! hardcoded to escape the bug
+```
+
+The failing construct there was `!$omp target loop private(SpV_dt)` in a `module subroutine` —
+**neither `pure` nor `do concurrent`**.
+
+## Workarounds, best to worst
+
+1. **Size the automatic from a dummy argument** (`size(h,3)`, or an `nz` dummy). Then nvfortran can
+   size it. *Untested — this is the cheap experiment to run before doing anything below.*
+2. **Hoist the array to the caller and `private()` it** on the directive — the `vert_friction:737`
+   pattern above. Known to work.
+3. **Compile-time `parameter`** (`nk=75`, `NK_GPU_MAX=500`). Works, but over-allocates every column
+   to the max: ~6.7x waste at a realistic `GV%ke≈75`, which will spill and crush occupancy.
+   Parameterize before merging anything that relies on it.
+4. **Static memory mode** sidesteps it entirely — `MOM_memory_macros.h:86` makes `SZK_(G)` expand to
+   `NK_`, a compile-time constant, versus `:172`'s `G%ke` in dynamic mode. **So this bug only bites
+   dynamic-memory builds.** Check which one you're on before chasing it.
+
+## Status
+
+- **Verified from source:** the `vert_friction:737` counter-example (merged + checksum-gated) and
+  the `05c74b56b` diff. No build/run was done.
+- **Inferred, untested:** that sizing from a dummy argument fixes it (workaround 1).
+- **Unclear:** which nvfortran versions are affected. Our evidence is 25.x; the
+  `porting-mom6-skill` skill claims 26.5 but may have inherited rather than re-tested the claim.
+
+## Note for the two existing write-ups
+
+Both are misleading and should be corrected against the table above:
+
+- **`porting-mom6-skill/SKILL.md`** says `do concurrent` cannot privatize an automatic array and to
+  *"fall back to `!$omp target teams loop`"*. But the failing construct was `target loop`, not DC —
+  and the suggested fallback is exactly what already works (`vert_friction:737`). It would send an
+  agent to swap constructs, which either succeeds for the wrong reason or hits the same wall.
+- **`KNOWLEDGE.md` §5 row 21** says *"in `pure`/DC procedures"* — but the evidence commit is neither.
+  (§9, "`NK_GPU_MAX=500` sizing", gets it right: "non-dummy-sized automatics", and already
+  recommends workaround 1.)
diff --git a/skills/gpu-data-residency/SKILL.md b/skills/gpu-data-residency/SKILL.md
@@ -0,0 +1,179 @@
+---
+name: gpu-data-residency
+description: Decide where `!$omp target enter data map(...)` belongs for a MOM6 array, choose `map(to:)` vs `map(alloc:)`, and find every place a device-resident array must be copied back to the host (`target update from`) or refreshed on device (`target update to`). Use when porting a module to GPU, adding a device-resident array, auditing a port for missing transfers or unbalanced enter/exit data, or debugging a checksum mismatch, answers-differ-by-GPU-count, or stale-host symptom. Triggers on "where do I map this", "enter data", "exit data", "update from", "copy back", "needed on the host", "residency", "stale host", "map(to) vs map(alloc)", "missing transfer", "map balance".
+---
+
+# GPU data residency: where to map, and where to copy back
+
+Answers two questions for a given array (or every array in a module):
+
+1. **Where does its `enter data`/`exit data` go, and with which map kind?**
+2. **Where must it be copied back** (`update from`) **or refreshed** (`update to`) **because a host-only consumer touches it?**
+
+Read `knowledge/KNOWLEDGE.md` §3 Step 5 (mapping lifecycle) and §3 Step 8 (transfer discipline) if not already in context.
+Depth: `knowledge/gpu-knowledge/03-openmp-mapping.md`, `12-diagnostics-io.md`, `02-pointer-usage.md`.
+
+## The engine: every mapped array is a two-copy shadow state
+
+Track **coherence between the host copy and the device copy** as you walk the code in execution
+order. Almost every mapping bug in this tree is a state-machine violation.
+
+| State | Host copy | Device copy |
+|---|---|---|
+| `UNMAPPED` | authoritative | does not exist |
+| `SYNCED` | valid | valid |
+| `HOST_FRESH` | authoritative | **stale / garbage** |
+| `DEV_FRESH` | **stale / garbage** | authoritative |
+
+| Event | Resulting state | Note |
+|---|---|---|
+| `enter data map(to: x)` | `SYNCED` | **Only if not already present.** On an already-present object this is a refcount bump and **copies nothing** — state unchanged (§8, "a `map(to:)` on an already-present object does not refresh device contents"). |
+| `enter data map(alloc: x)` | `HOST_FRESH` | Device side is garbage. Legal only if the next device touch is a *write*. |
+| host write | `HOST_FRESH` | includes `!$OMP parallel do` loops — those are **host** CPU threads |
+| device write | `DEV_FRESH` | |
+| **host read while `DEV_FRESH`** | **BUG** | insert `!$omp target update from(x)` before it |
+| **device read while `HOST_FRESH`** | **BUG** | insert `!$omp target update to(x)` before it |
+| `update from(x)` | `SYNCED` | device → host |
+| `update to(x)` | `SYNCED` | host → device |
+| `exit data map(from: x)` | `UNMAPPED`, host valid | copies back |
+| `exit data map(delete:/release: x)` | `UNMAPPED`, host **as it was** | **neither kind copies back** |
+
+The machine tracks *coherence*, not *initialization*: `HOST_FRESH` on a freshly-`map(alloc:)`'d
+local automatic array means "host is authoritative and holds garbage". Both reads are still wrong;
+flag them separately.
+
+Two rules the mechanical walk will not derive on its own:
+
+- **`delete` vs `release`** — `delete` forces the refcount to **zero**, destroying any *outer*
+  persistent mapping of the same object. Use `release` for scoped/per-call teardown; `delete` only
+  in the `*_end` that mirrors the owning `enter data`. Never `map(delete:)` an object your scope
+  does not own (`vertvisc`'s `map(delete: ADp)` silently kills `initialize_MOM`'s map — §8, "the `ADp` mapping lifecycle").
+- **Re-mapping never refreshes.** If host scalars/descriptors changed after the first map, the only
+  refresh is `target update to(...)`. Never "re-map to refresh"; never re-`enter data` a parent
+  struct after its members are attached (`c82e1254a`).
+
+## Procedure
+
+### Step 1 — Scope and inventory
+
+Pick the array(s). For each, get every touch site interleaved with every region marker, in line
+order:
+
+```bash
+scripts/residency-scan.sh <file.F90> <array-name>
+```
+
+It tags each line `MAP` / `XFER` / `DEV-REGION` / `DEV-HALO` / `HOST-THREADS` / `HOST-SINK` /
+`TOUCH`. It is a *reading aid*, not an oracle — it tags the lines that open regions, and you still
+have to read the code to see which touches fall inside them.
+
+Also establish the array's identity, which fixes where the map goes:
+
+| Kind | `enter data` site | `exit data` site |
+|---|---|---|
+| CS member (`ALLOCABLE_`) | in `<mod>_init`, next to `ALLOC_`, after `... = 0.0` | in `<mod>_end`, next to `DEALLOC_`, mirrored member-by-member, `delete` |
+| Subroutine-scope scratch | at routine entry, `map(alloc:)` | at return, `release` (early-release once last use passes is fine) |
+| Dummy argument | **not here** — the caller owns it; verify residency at every call site | — |
+| Pointer member | `map(to:)` **never `alloc`**, guarded `if (associated(x))` | mirrored, same guard |
+
+### Step 2 — Classify every touch HOST or DEVICE
+
+This is where the analysis is won or lost. Reason about the **GPU build**
+(`__NVCOMPILER_OPENMP_GPU`).
+
+| Marker | Verdict |
+|---|---|
+| `do concurrent (...)` | **DEVICE** — the default compute idiom |
+| `!$omp target teams` / `!$omp target ... loop` | **DEVICE** |
+| `!$OMP parallel do` / `!$omp parallel` | **HOST** — CPU threads. *The single most common misread.* |
+| plain `do` loop | **HOST** |
+| plain `do` loop **inside** a `target teams loop` / DC | **DEVICE** (serial-k columns) |
+| call to a `pure`/`elemental`/`declare target` helper from a device region | **DEVICE** |
+| any other `call` | **HOST** unless proven otherwise |
+
+Then check the touch against the **host-boundary catalogue** — the calls that are host-only and
+therefore force a copy-back. See `references/host-boundaries.md` for the full list with its
+verification commands. The load-bearing ones:
+
+- `post_data` and the whole diag mediator; `hchksum`/`uvchksum`/... ; `save_restart`
+- `pass_var`/`pass_vector` (no `omp_offload` argument exists); `start_group_pass`/`complete_group_pass`
+- `do_group_pass(..., omp_offload=.true.)` is **DEVICE** — no transfer. Without the flag: HOST.
+- any call into an untouched module (diabatic stack, ALE remap/regrid, restart — `knowledge/KNOWLEDGE.md` §2.4)
+
+### Step 3 — Walk the ledger
+
+In execution order, one row per event. **Branches matter more than anything else here**: a transfer
+inside `if (cond)` does not dominate a read outside it. When a device write and a host read sit in
+sibling branches, write down the predicate that reaches the read without the transfer — that
+conjunction *is* the bug report.
+
+| # | Line | Event | Host/Dev | State after | Verdict |
+|---|---|---|---|---|---|
+| 1 | `:209` | `map(alloc: khdt_x)` | — | `HOST_FRESH` (garbage) | ok |
+| 2 | `:291` | write in `do concurrent` | DEV | `DEV_FRESH` | ok |
+| 3 | `:394` | read in plain `do` | HOST | — | **BUG: needs `update from`** |
+
+### Step 4 — Emit directives
+
+Map kind, from the *first device touch* and who else reads the contents:
+
+- first device touch is a **read**, or any host-set scalar/pointer descriptor is read on device
+  → **`map(to:)`**. This includes every struct whose `associated()` state feeds device control flow
+  (`map(alloc: Reg, Reg%Tr(:))` was the multi-GPU answer-change bug, `a774eb331`).
+- first device touch is a **write**, pure workspace → **`map(alloc:)`**.
+- CS shells → `map(alloc:)`, mapped **once**, before the child `_init`.
+
+Transfer placement:
+
+- Put the transfer at the **producer**, immediately upstream of the consumer. A transfer of a
+  *different* array does not cover yours (`b29b27150`).
+- **Decouple transfer from post**: one `update from` covering all consumers, guarded
+  `if (CS%debug .or. CS%id_a>0 .or. CS%id_b>0)`, then the individual `if (id>0) call post_data(...)`
+  (`MOM_diagnostics.F90:1825-1827`).
+- Bracket an unavoidable host-only detour both ways: `from(...)` before, `to(...)` after
+  (ALE at `MOM.F90:1036/1038`).
+- At coarse sync points a blanket transfer is the codebase default (`MOM.F90:1091`) — match it
+  unless profiling shows a stall.
+- Guard with the matching intrinsic: `if (associated(x))` for pointers, `if (allocated(x))` for
+  allocatables — never mixed. Never `map(...) if (present(optional))` inside a callee
+  (`2108e0eba`).
+
+### Step 5 — Lifecycle and balance checks
+
+Run these over the routine/module regardless of what the walk found:
+
+1. **Every `enter data` has a mirrored `exit data`** in the same scope (`15ca2a25f` leaked
+   `b_denom_1`).
+2. **No `delete` on an object this scope does not own** (§8, "the `ADp` mapping lifecycle").
+3. **No copy-back expected from `delete`/`release`** — if the host needs the value, an `update from`
+   or `map(from:)` must precede it.
+4. **Parent mapped exactly once**, members attached after, refreshed with `update to(parent)`.
+5. **Restart-registered, device-mutated fields** have a dominating `update from` before
+   `save_restart` (currently latent-only — §8, "restart staleness is latent, not live"; do not let your port break it).
+6. **Arrays-of-structs are not mapped element-by-element** on a hot path (`1865612de`).
+
+### Step 6 — Report
+
+Lead with the verdict. For each finding give: array, the two sites (device write → host read), the
+**predicate that reaches the bad read**, the symptom it would produce, and the one-line fix. Then
+the proposed directive set. Separate **confirmed** (you read both sites and the branch structure)
+from **suspected** (needs a run).
+
+## Symptoms this analysis explains
+
+| Symptom | Likely residency cause |
+|---|---|
+| Checksum "mismatch" that isn't reproducible arithmetic | host-only checksum read a stale host copy — missing `update from` (`b29b27150`) |
+| Answers differ **by GPU count** / run-to-run, control flow | `map(alloc:)` on a struct whose host-set scalars or `associated()` are read on device (`a774eb331`) |
+| Device "addressing error" after init; `associated()` misbehaves | parent re-`enter data`'d after members attached (`c82e1254a`) |
+| Correct on 1 GPU, wrong on N | missing `reduce`, or `alloc`-vs-`to` — latent until multi-device |
+| Silent garbage in a host diagnostic only in some configs | transfer guarded by a *different* predicate than the read (the `khdt_x` shape — see `references/worked-example.md`) |
+| GPU time dominated by attach/detach | array-of-structs mapped per element (`1865612de`) |
+| Unexplained per-statement traffic | array touched in a device region with no explicit map at all (`bc05a6a89`) |
+
+## Verification gate
+
+A residency fix is a correctness change: it is subject to the same gate as any port
+(`knowledge/KNOWLEDGE.md` §3 Step 9) — bit-identical `MOM_checksums` field checksums plus EFP
+`write_energy`, and a **≥2-GPU run**, because the `alloc`-vs-`to` class of bug is latent on one
+device. Never accept a nonzero diff as rounding.