Skip to content

workbench_idemix2: IDEMIX2 - Internal Wave Energy Parameterisation for FESOM2 - #927

Open
patrickscholz wants to merge 90 commits into
mainfrom
workbench_idemix2
Open

workbench_idemix2: IDEMIX2 - Internal Wave Energy Parameterisation for FESOM2#927
patrickscholz wants to merge 90 commits into
mainfrom
workbench_idemix2

Conversation

@patrickscholz

Copy link
Copy Markdown
Contributor

IDEMIX2: Internal Wave Energy Parameterisation for FESOM2

IDEMIX2 (Internal wave Dissipation, Energy and MIXing, version 2) is a prognostic parameterisation of internal gravity wave energy and its contribution to diapycnal mixing, based on Olbers & Eden (2013) and Eden et al. (2014). It is implemented here as an extension to the existing IDEMIX1/TKE mixing scheme in FESOM2.

Concept

Unlike IDEMIX1, which tracks a single vertically integrated internal wave energy field, IDEMIX2 resolves the horizontal propagation direction of internal waves through a spectral angular dimension (nfbin directional bins, default 50 interior + 2 ghost bins covering 0–2π). This captures anisotropic wave fields near generation sites such as rough topography and coastlines, and allows wave energy to be tracked as it propagates across ocean basins.

Wave energy compartments

Two spectral wave energy fields are evolved independently:

  Surface wind forcing                 M2 tidal generation
  (NIW, near-inertial)                 (rough topography)
          │                                    │
          ▼                                    ▼
  ┌───────────────┐                  ┌─────────────────┐
  │  E_niw(φ, x)  │                  │   E_M2(φ, x)    │
  │  Near-inertial│                  │   M2 tidal IW   │
  │  waves        │                  │   (~12.4 h)     │
  └───────┬───────┘                  └────────┬────────┘
          │  ω ≈ f (lat-dependent)            │  ω = ω_M2 (fixed)
          └──────────────┬────────────────────┘
                         │  wave-wave interaction
                         ▼
               ┌──────────────────┐
               │   E_iw(z, x)     │   background IW continuum
               │   (vertically    │   (GM spectrum)
               │    resolved)     │
               └────────┬─────────┘
                        │  dissipation → diapycnal mixing
                        ▼
               ┌──────────────────┐
               │   TKE scheme     │   iw_diss added as
               │   Kv, Av         │   energy source term
               └──────────────────┘

Timestep structure

Each call to calc_cvmix_idemix2 follows this sequence:

  ┌─────────────────────────────────────────────────────────────────┐
  │ 1. Node parameters                                              │
  │    cn (baroclinic wave speed), α_c, v0, c0, τ_M2, τ_niw,        │
  │    vertical structure functions  [loop: node, OMP parallel]     │
  ├─────────────────────────────────────────────────────────────────┤
  │ 2. Element group velocities                                     │
  │    u_M2(φ), v_M2(φ), w_M2(φ)  — spectral propagation speeds     │
  │    scatter elem → node (area-weighted)                          │
  ├─────────────────────────────────────────────────────────────────┤
  │ 3. Horizontal spectral integration  [for M2, then NIW]          │
  │                                                                 │
  │   E^n  ──▶  superbee horizontal advection  ──▶  flx_uv(φ,edge)  │
  │              │                                                  │
  │              ├──▶  reflective coastal BC                        │
  │              │     (redirect toward-coast flux into             │
  │              │      mirror spectral bin at interior node)       │
  │              ▼                                                  │
  │         superbee cross-spectral advection ──▶  flx_cs(φ,node)   │
  │              │                                                  │
  │              ▼                                                  │
  │         flux → divergence  +  reflect_src injection             │
  │              │                                                  │
  │              ▼                                                  │
  │         E^(n+1) = (E^n + dt·(Edivh + Edivs + forc))             │
  │                   / (1 + dt·τ)     [AB2 or Euler]               │
  ├─────────────────────────────────────────────────────────────────┤
  │ 4. Eiw vertical diffusion + implicit dissipation                │
  │    tridiagonal solve per node  [OMP parallel]                   │
  ├─────────────────────────────────────────────────────────────────┤
  │ 5. Eiw horizontal diffusion  (optional)                         │
  ├─────────────────────────────────────────────────────────────────┤
  │ 6. Wave-wave interaction  E_M2/niw ↔ E_iw                       │
  │    [loop: node, OMP parallel]                                   │
  ├─────────────────────────────────────────────────────────────────┤
  │ 7. iw_diss → TKE (Kv, Av)                                       │
  └─────────────────────────────────────────────────────────────────┘

Reflective coastal boundary condition

Wave energy reaching a coastline is redirected into the spectrally mirrored propagation direction at the last interior node, rather than leaking into land:

  LAND │ n_coast ──── edge ──── n_int (interior node)
       │
       │  toward-coast flux in bin φ_i  →  zeroed
       │                                       │
       │                                       ▼
       │                            energy re-injected at n_int
       │                            in mirror bin φ_k = edge normal
       │                            (net energy change = 0,
       │                             spectral redistribution only)

The mirror bin φ_k for each coast edge is pre-computed once at init from the edge geometry and stored in iwe2_refl_bin(edge).

Forcing inputs

Field Source Role
M2 tidal forcing Observed anisotropic tidal generation (modes 1+2) E_M2 surface source
NIW / surface forcing Wind-derived near-inertial flux E_niw surface source or E_iw surface BC
Bottom tidal forcing Near-field tidal dissipation dataset E_iw bottom source
Topographic HRMS / λ Seafloor roughness fields Scattering timescale τ

patrickscholz and others added 30 commits January 28, 2026 16:23
…sages if something is wrong in the forcing files
…ested yet, it is discarded for the moment for the idemix2 implementation
…ter, fix some bugs in the horizontal advection of E_niw
…e diagnostic

IDEMIX2 changes across five files:

namelist / cvmix_idemix2.F90
- Add four new namelist parameters (all timescales in days, larger = slower
  dissipation = more open-ocean energy):
    idemix2_tau_M2_shelf  = 7    (M2 shelf dissipation timescale)
    idemix2_tau_M2_oce    = 50   (M2 open-ocean floor timescale)
    idemix2_tau_niw_shelf = 7    (NIW shelf dissipation timescale)
    idemix2_tau_niw_oce   = 3    (NIW open-ocean floor timescale)
- Wire them into cvmix_idemix2_setup as optional dummy args with defaults
- Replace hard-coded magic numbers in tau_compart with the new namelist
  values; separate shelf and open-ocean branches for M2 vs NIW

gen_modules_cvmix_idemix2.F90
- Shrink all IO-only diagnostic arrays from node_size (local+halo) to
  myDim_nod2D (local only): iwe2_E_iw_fbot, iwe2_E_iw_vdif, iwe2_E_iw_hdif,
  iwe2_E_iw_fsrf, iwe2_E_iw_dt; and the M2/NIW Ecompart/WWI arrays
  (iwe2_E_M2_advh/advs/diss/forc/refl/dt, iwe2_E_iw_diss_M2,
  iwe2_E_M2_diss_wwi, NIW equivalents)
- Add `node <= myDim_nod2D` guard in the Eiw vertical diffusion loop and
  all three WWI branches so diagnostic arrays are never written at halo
  indices (fixes latent OOB for the M2/NIW WWI arrays)
- Fix iwe2_E_iw_hdif whole-array assignment to restrict RHS to 1:myDim_nod2D
- Add iwe2_coast_node (logical, node_size): marks nodes ON the coast
  boundary. Set in init_reflect_bc via a real-valued proxy to allow
  exchange_nod to propagate coast status across MPI halo boundaries.
  Distinguishes coast nodes from iwe2_refl_node (coast-adjacent interior).
- Fix ivols_int missing from !$OMP PARALLEL DO PRIVATE list in
  apply_reflect_bc_spctrl (thread-safety fix)
- Fix integer division 1/vol_s -> 1.0_WP/vol_s
- Correct hsintegrate_Ecompart optional dummy arg sizes to myDim_nod2D;
  iwe2_E_M2_refl/iwe2_E_niw_refl changed from 2D (nfbin, myDim_nod2D)
  to 1D (myDim_nod2D) matching the scalar-per-node reflected-flux design

io_meandata.F90
- Update def_stream calls for iwe2_Em2_refl and iwe2_Eniw_refl to 1D
  node-field form, matching the redesigned scalar storage

io_mesh_info.F90
- Write iwe2_coast_node to mesh.diag.nc alongside iwe2_refl_node
@patrickscholz
patrickscholz marked this pull request as ready for review June 24, 2026 10:29
@JanStreffing

Copy link
Copy Markdown
Collaborator

Nice work!
Should we add a CI test for IDEMIX2?

@JanStreffing

Copy link
Copy Markdown
Collaborator

I need some time to check this. It's quite a bit. Beginning of next week ok?

@JanStreffing JanStreffing added this to the FESOM 2.8 milestone Jun 25, 2026
patrickscholz and others added 5 commits June 30, 2026 15:03
…rame

  Two related fixes after investigating pyOM2's spherical-coordinate treatment
  of the spectral group velocity:

  cvmix_idemix2.F90 -- v_compart: drop coslat, replace tentative comment
    pyOM2 multiplies v_M2 by cosu(j) because on a lat-lon B-grid the V-face
    arc-length is r*cos(lat_face)*dx, so the cell-budget reads
    [cosu(j)*v*E(j) - cosu(j-1)*v*E(j-1)] / (cost(j)*dyt(j)), which is the
    correct discrete form of (1/r cos phi) d(cos phi * F_y)/dphi. In FESOM2's
    unstructured FV scheme the cos(lat) metric is already embedded in the
    physical edge lengths (via elem_cos) and the true nodal control-volume
    vol_s; adding an extra coslat factor on v_compart would double-count the
    metric and bias meridional transport by O(cos lat). FESOM2 standard
    tracer advection applies the same convention, so omitting coslat here is
    consistent. The old placeholder comment is replaced with this explanation.

  gen_modules_cvmix_idemix2.F90 -- init_reflect_bc: use coord_nod2D not geo_coord_nod2D
    The mirror spectral bin for a coast edge is found by computing the angle
    from coast node to interior node and matching it to phit(k). phit lives
    in the rotated model frame because u_compart = cg*cos(phit) is projected
    onto edge_cross_dxdy, which is built from coord_nod2D with the rotated-
    latitude cos metric. Using geographic coordinates geo_coord_nod2D
    introduced a misalignment between the reflected bin and the actual
    propagation direction in non-trivial mesh rotations.
Conflicts were all between the idemix2 timing rework and the RECOM
task-parallel (__recom && __usetp) support that landed on main. Both are
kept: idemix2's finer timers, main's group-0 guards and RECOM interfaces.

src/fesom_module.F90 (13 conflicts)
  - keep idemix2's named timers (t_ice_*, t_oce_*, t_diag_*, t_io_*,
    t_rstart_*, t_iceb_*) and the finer ice breakdown; drop main's
    t0_ice/t1_ice/t0_frc/t1_frc/rtime_fullice/rtime_read_forcing, which
    have no remaining references
  - keep main's __usetp pointers (my_fesom_group, MPI_COMM_FESOM_WORLD,
    MPI_COMM_FESOM_SAME_RANK_IN_GROUPS) and every group-0 guard
  - keep main's extended REcoM call signatures (use_age_tracer,
    use_transit, l_sf6, l_f11, l_f12, l_r14c, l_r39ar, ocean_area)
  - timers are placed outside the guards, so every group times the work
    it actually executes
  - keep main's loop counter i in fesom_finalize alongside idemix2's
    allocatable mean/max/min_rtime; it is used by the __usetp group loop
  - fix a sign slip in the REcoM timer accumulation: the rename had left
    t_recom_s - t_recom_e, which accumulated negative time

src/oce_ale.F90 (4 conflicts)
  - keep idemix2's t11 numbering and the rtime_oce_presdens /
    rtime_oce_mixing split of main's rtime_oce_mixpres
  - keep main's oce_bt_rhs / oce_bt_step profiler regions and the removal
    of the degenerate empty oce_hbar_calc pair in the split-explicit
    branch; its comment updated to the idemix2 numbering (t6=t5)
  - keep main's group-0 guard around the step-timing printout

src/io_restart.F90
  - main's collective tracer valuesold/valuesAB Bcast stays outside the
    group-0 guard; prepare_idemix2_restart() goes inside it, before the
    ocean restart write

src/io_meandata.F90
  - keep idemix2's is_fbin_based and main's flip = .false. initialisation
The merge resolution appended 'integer :: i' after the allocate() of the
rtime arrays, which is an executable statement, so the declaration landed
in the executable section:

  error #6236: A specification statement cannot appear in the executable section
  error #6404: This name does not have a type, and must have an explicit type. [I]

Move the declaration above the allocate. Build with -DRECOM_COUPLED=ON
(which also sets __usetp) now compiles and links.
namelist.cvmix carried

  idemix2_AB_epsilon    = 0.1_WP

'0.1_WP' is Fortran *source* syntax; a NAMELIST input record cannot carry a
kind suffix. Any run selecting an IDEMIX2 scheme (cvmix_IDEMIX2 or
cvmix_TKE+cvmix_IDEMIX2) therefore aborted while reading the namelist,
before the first timestep:

  forrtl: severe (17): syntax error in NAMELIST input, unit 20,
  file namelist.cvmix, line 95, position 28
    ... g_cvmix_idemix2_m -> ocean_setup_ -> fesom_module

Write it as a plain literal. It is the only kind-suffixed value in any
shipped namelist.
@JanStreffing

Copy link
Copy Markdown
Collaborator

Tested this branch on levante (pi mesh, 1 day, 32 steps/day, cvmix_TKE+cvmix_IDEMIX2).

Namelist blocker, already fixed on the branch in 7ad2944. config/namelist.cvmix had idemix2_AB_epsilon = 0.1_WP. The _WP kind suffix is Fortran source syntax and is not legal in a namelist record, so every run selecting an IDEMIX2 scheme aborted while reading the namelist, before the first timestep:

forrtl: severe (17): syntax error in NAMELIST input, unit 20,
file namelist.cvmix, line 95, position 28
  ... g_cvmix_idemix2_m -> ocean_setup_ -> fesom_module

It was the only kind-suffixed value in any shipped namelist, so IDEMIX2 could not have been run from the shipped config before this.

Could we get the IDEMIX2 forcing staged on levante? All five IDEMIX2 forcing entries point into /albedo/work/user/pscholz/... or /albedo/pool/..., so none resolve here. There are two separate cases. idemix2_niwforc_file is already present on levante at /pool/data/AWICM/FESOM2/FORCING/IDEMIX/fourier_smooth_2005_cfsr_inert_rgrid.nc under the same filename, so that one only needs the namelist path corrected, the way idemix_surforc_file a few lines above already does it. The M2, HRMS, lambda and STORMTIDE2 bottom-forcing files exist only on albedo and would need copying into /pool/data/AWICM/FESOM2/FORCING/. Once they are there the shipped namelist could point at paths that work on both machines.

Note that the NIW file is read even with idemix2_enable_niw = .false., which the error message states explicitly, so it is a hard dependency for any IDEMIX2 run.

With the NIW path pointed at the levante copy and idemix2_enable_bot = .false., IDEMIX2 runs clean, standalone on 2 ranks and alongside REcoM on 4. iwe2_E_iw starts at zero and after one day is populated over roughly 70% of wet cells with a max of 1.2e-2 m2/s2, so the E_iw evolution, vertical diffusion, dissipation and TKE coupling are all doing work.

That is as far as we can take it from here though. The bottom tidal source and the E_M2 and E_niw compartments cannot be exercised at all without the forcing files, so IDEMIX2 cannot be properly tested on levante until they are staged.

@JanStreffing

Copy link
Copy Markdown
Collaborator

@patrickscholz, @koldunovn Can perhaps update /pool/data/AWICM/FESOM2/FORCING/IDEMIX, or create /pool/data/AWICM/FESOM2/FORCING/IDEMIX2

…bels

The timing-table rework renumbered the format statements: 43 changed from
(a33,i15) to a runtime-percentage format, the integer format moved to
44/45 and the runtime format to 46. The SSH CG solver summary still
printed its integers with 'print 43' and its mean with 'print 45', which
aborts every run at finalization with

    Fortran runtime error: Expected REAL for item 2 in formatted
    transfer, got INTEGER

This is the failure behind all red CI jobs on this PR.
# Conflicts:
#	src/fesom_module.F90
#	src/gen_modules_read_NetCDF.F90
#	src/io_meandata.F90
#	src/oce_tracer_mod.F90
#	src/write_step_info.F90
Buffers declared real(kind=WP) were communicated as MPI_DOUBLE_PRECISION:
the read_other_NetCDF/-_3d broadcasts (which also serve the pre-existing
runoff/SSS/chlorophyll/ice readers), the IDEMIX2 init/diagnostic
Allreduces (one of which normalizes the bottom tidal forcing), the fbin
halo-exchange dummies (declared real64) and the fbin MPI type creation.
In a USE_SINGLE_PRECISION build (WP=4) each such transfer reads/writes
twice the buffer - heap corruption on the broadcasts, garbage physics in
the forcing normalization. Use MPI_WP and real(kind=WP) throughout, as
main does since the single-precision support landed; double-precision
builds are unchanged. The fbin gather-to-master routines keep real64,
matching the convention of the neighbouring gather routines.
…tep reads

apply_idemix2_restart moved the restart-saved divergence from slot 1 to
slot 2 and zeroed slot 1. But the first step after restart swaps to
ti=2/tip1=1: the fresh divergence is computed into slot 2 (clobbering the
saved values unread) and the AB2 'previous divergence' is read from slot
1 (just zeroed). The saved divh/divs restart fields were therefore never
used and the first post-restart step degraded to an over-weighted Euler
step, making restarts non-reproducible against a continuous run. Keep
the saved divergence in slot 1 and zero slot 2 instead; the E-field
staging was already correct.
cvmix_IDEMIX2 took digit 7 and cvmix_TIDAL moved to digit 8, but only
cvmix_PP+cvmix_TIDAL (47->48) and the oce_ale branch checks were
renumbered. KPP+cvmix_TIDAL, PP+cvmix_TIDAL and cvmix_KPP+cvmix_TIDAL
still mapped to 17/27/37: their base scheme never ran (oce_ale checks
18/28/38) and mod(...,10)==7 routed them into IDEMIX2 instead of TIDAL.
Move the three mappings to 18/28/38 and update every consumer
(oce_setup_step, oce_ale_tracer, gen_modules_diag, io_meandata,
oce_ale_pressure_bv) to match.
…e-path ORDERED region

The spectral flux-to-tracer edge loop emits an !$OMP ORDERED region on
the __openmp_reproducible path but its !$OMP DO lacked the required
ORDERED clause - the same defect class PR #978 eliminates tree-wide, and
its ctest guard flags this file the moment both merge. Guard the clause
so the default OpenMP (lock-based) build is byte-identical. Verified
with PR #978's check_omp_ordered.py: this file now passes both the
reproducible and default invariants.
areasvol is zero above ulevels_nod2D: for a node under an ice-shelf
cavity (ulevels>1) areasvol(1,node) is exactly 0, so the element-to-node
averages in interp_e2n_1d/2d/3d divide by zero the moment IDEMIX2 runs
on a cavity mesh (Inf/NaN, or an FPE abort). Use the area of the node's
first wet level; identical on non-cavity meshes where ulevels==1.
…ision WP

The module hard-coded its kind to the external library's cvmix_r8
(double). In a USE_SINGLE_PRECISION build the driver's real(WP)=real32
fields then match no specific of the generic interfaces and the build
fails ('There is no specific subroutine for the generic
cvmix_idemix2_compute_...', 19 call sites) - exposed by sp_ctests once
main was merged. Unlike the external CVMix library (kept double behind a
conversion boundary, cf. gen_modules_cvmix_tke/idemix), this module is
FESOM-owned: bind its kind to WP (keeping the cvmix_r8 name to avoid
touching every declaration) so all driver calls match in both builds.
The single call into the fixed real*8 library helper solve_tridiag now
converts at its call site; double-precision builds are bit-identical
throughout.
@JanStreffing

Copy link
Copy Markdown
Collaborator

@patrickscholz: Adapted to current state of main branch. Mostly regarding single precision, and some omp ordering. Please take a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants