workbench_idemix2: IDEMIX2 - Internal Wave Energy Parameterisation for FESOM2 - #927
workbench_idemix2: IDEMIX2 - Internal Wave Energy Parameterisation for FESOM2#927patrickscholz wants to merge 90 commits into
Conversation
…sages if something is wrong in the forcing files
…ested yet, it is discarded for the moment for the idemix2 implementation
…ter, fix some bugs in the horizontal advection of E_niw
…com/FESOM/fesom2 into origin/tmp/apply_patches_for_icb_v3
…e diagnostic
IDEMIX2 changes across five files:
namelist / cvmix_idemix2.F90
- Add four new namelist parameters (all timescales in days, larger = slower
dissipation = more open-ocean energy):
idemix2_tau_M2_shelf = 7 (M2 shelf dissipation timescale)
idemix2_tau_M2_oce = 50 (M2 open-ocean floor timescale)
idemix2_tau_niw_shelf = 7 (NIW shelf dissipation timescale)
idemix2_tau_niw_oce = 3 (NIW open-ocean floor timescale)
- Wire them into cvmix_idemix2_setup as optional dummy args with defaults
- Replace hard-coded magic numbers in tau_compart with the new namelist
values; separate shelf and open-ocean branches for M2 vs NIW
gen_modules_cvmix_idemix2.F90
- Shrink all IO-only diagnostic arrays from node_size (local+halo) to
myDim_nod2D (local only): iwe2_E_iw_fbot, iwe2_E_iw_vdif, iwe2_E_iw_hdif,
iwe2_E_iw_fsrf, iwe2_E_iw_dt; and the M2/NIW Ecompart/WWI arrays
(iwe2_E_M2_advh/advs/diss/forc/refl/dt, iwe2_E_iw_diss_M2,
iwe2_E_M2_diss_wwi, NIW equivalents)
- Add `node <= myDim_nod2D` guard in the Eiw vertical diffusion loop and
all three WWI branches so diagnostic arrays are never written at halo
indices (fixes latent OOB for the M2/NIW WWI arrays)
- Fix iwe2_E_iw_hdif whole-array assignment to restrict RHS to 1:myDim_nod2D
- Add iwe2_coast_node (logical, node_size): marks nodes ON the coast
boundary. Set in init_reflect_bc via a real-valued proxy to allow
exchange_nod to propagate coast status across MPI halo boundaries.
Distinguishes coast nodes from iwe2_refl_node (coast-adjacent interior).
- Fix ivols_int missing from !$OMP PARALLEL DO PRIVATE list in
apply_reflect_bc_spctrl (thread-safety fix)
- Fix integer division 1/vol_s -> 1.0_WP/vol_s
- Correct hsintegrate_Ecompart optional dummy arg sizes to myDim_nod2D;
iwe2_E_M2_refl/iwe2_E_niw_refl changed from 2D (nfbin, myDim_nod2D)
to 1D (myDim_nod2D) matching the scalar-per-node reflected-flux design
io_meandata.F90
- Update def_stream calls for iwe2_Em2_refl and iwe2_Eniw_refl to 1D
node-field form, matching the redesigned scalar storage
io_mesh_info.F90
- Write iwe2_coast_node to mesh.diag.nc alongside iwe2_refl_node
|
Nice work! |
|
I need some time to check this. It's quite a bit. Beginning of next week ok? |
…rame
Two related fixes after investigating pyOM2's spherical-coordinate treatment
of the spectral group velocity:
cvmix_idemix2.F90 -- v_compart: drop coslat, replace tentative comment
pyOM2 multiplies v_M2 by cosu(j) because on a lat-lon B-grid the V-face
arc-length is r*cos(lat_face)*dx, so the cell-budget reads
[cosu(j)*v*E(j) - cosu(j-1)*v*E(j-1)] / (cost(j)*dyt(j)), which is the
correct discrete form of (1/r cos phi) d(cos phi * F_y)/dphi. In FESOM2's
unstructured FV scheme the cos(lat) metric is already embedded in the
physical edge lengths (via elem_cos) and the true nodal control-volume
vol_s; adding an extra coslat factor on v_compart would double-count the
metric and bias meridional transport by O(cos lat). FESOM2 standard
tracer advection applies the same convention, so omitting coslat here is
consistent. The old placeholder comment is replaced with this explanation.
gen_modules_cvmix_idemix2.F90 -- init_reflect_bc: use coord_nod2D not geo_coord_nod2D
The mirror spectral bin for a coast edge is found by computing the angle
from coast node to interior node and matching it to phit(k). phit lives
in the rotated model frame because u_compart = cg*cos(phit) is projected
onto edge_cross_dxdy, which is built from coord_nod2D with the rotated-
latitude cos metric. Using geographic coordinates geo_coord_nod2D
introduced a misalignment between the reflected bin and the actual
propagation direction in non-trivial mesh rotations.
Conflicts were all between the idemix2 timing rework and the RECOM
task-parallel (__recom && __usetp) support that landed on main. Both are
kept: idemix2's finer timers, main's group-0 guards and RECOM interfaces.
src/fesom_module.F90 (13 conflicts)
- keep idemix2's named timers (t_ice_*, t_oce_*, t_diag_*, t_io_*,
t_rstart_*, t_iceb_*) and the finer ice breakdown; drop main's
t0_ice/t1_ice/t0_frc/t1_frc/rtime_fullice/rtime_read_forcing, which
have no remaining references
- keep main's __usetp pointers (my_fesom_group, MPI_COMM_FESOM_WORLD,
MPI_COMM_FESOM_SAME_RANK_IN_GROUPS) and every group-0 guard
- keep main's extended REcoM call signatures (use_age_tracer,
use_transit, l_sf6, l_f11, l_f12, l_r14c, l_r39ar, ocean_area)
- timers are placed outside the guards, so every group times the work
it actually executes
- keep main's loop counter i in fesom_finalize alongside idemix2's
allocatable mean/max/min_rtime; it is used by the __usetp group loop
- fix a sign slip in the REcoM timer accumulation: the rename had left
t_recom_s - t_recom_e, which accumulated negative time
src/oce_ale.F90 (4 conflicts)
- keep idemix2's t11 numbering and the rtime_oce_presdens /
rtime_oce_mixing split of main's rtime_oce_mixpres
- keep main's oce_bt_rhs / oce_bt_step profiler regions and the removal
of the degenerate empty oce_hbar_calc pair in the split-explicit
branch; its comment updated to the idemix2 numbering (t6=t5)
- keep main's group-0 guard around the step-timing printout
src/io_restart.F90
- main's collective tracer valuesold/valuesAB Bcast stays outside the
group-0 guard; prepare_idemix2_restart() goes inside it, before the
ocean restart write
src/io_meandata.F90
- keep idemix2's is_fbin_based and main's flip = .false. initialisation
The merge resolution appended 'integer :: i' after the allocate() of the rtime arrays, which is an executable statement, so the declaration landed in the executable section: error #6236: A specification statement cannot appear in the executable section error #6404: This name does not have a type, and must have an explicit type. [I] Move the declaration above the allocate. Build with -DRECOM_COUPLED=ON (which also sets __usetp) now compiles and links.
namelist.cvmix carried
idemix2_AB_epsilon = 0.1_WP
'0.1_WP' is Fortran *source* syntax; a NAMELIST input record cannot carry a
kind suffix. Any run selecting an IDEMIX2 scheme (cvmix_IDEMIX2 or
cvmix_TKE+cvmix_IDEMIX2) therefore aborted while reading the namelist,
before the first timestep:
forrtl: severe (17): syntax error in NAMELIST input, unit 20,
file namelist.cvmix, line 95, position 28
... g_cvmix_idemix2_m -> ocean_setup_ -> fesom_module
Write it as a plain literal. It is the only kind-suffixed value in any
shipped namelist.
|
Tested this branch on levante (pi mesh, 1 day, 32 steps/day, Namelist blocker, already fixed on the branch in 7ad2944. It was the only kind-suffixed value in any shipped namelist, so IDEMIX2 could not have been run from the shipped config before this. Could we get the IDEMIX2 forcing staged on levante? All five IDEMIX2 forcing entries point into Note that the NIW file is read even with With the NIW path pointed at the levante copy and That is as far as we can take it from here though. The bottom tidal source and the E_M2 and E_niw compartments cannot be exercised at all without the forcing files, so IDEMIX2 cannot be properly tested on levante until they are staged. |
|
@patrickscholz, @koldunovn Can perhaps update /pool/data/AWICM/FESOM2/FORCING/IDEMIX, or create /pool/data/AWICM/FESOM2/FORCING/IDEMIX2 |
…bels
The timing-table rework renumbered the format statements: 43 changed from
(a33,i15) to a runtime-percentage format, the integer format moved to
44/45 and the runtime format to 46. The SSH CG solver summary still
printed its integers with 'print 43' and its mean with 'print 45', which
aborts every run at finalization with
Fortran runtime error: Expected REAL for item 2 in formatted
transfer, got INTEGER
This is the failure behind all red CI jobs on this PR.
# Conflicts: # src/fesom_module.F90 # src/gen_modules_read_NetCDF.F90 # src/io_meandata.F90 # src/oce_tracer_mod.F90 # src/write_step_info.F90
Buffers declared real(kind=WP) were communicated as MPI_DOUBLE_PRECISION: the read_other_NetCDF/-_3d broadcasts (which also serve the pre-existing runoff/SSS/chlorophyll/ice readers), the IDEMIX2 init/diagnostic Allreduces (one of which normalizes the bottom tidal forcing), the fbin halo-exchange dummies (declared real64) and the fbin MPI type creation. In a USE_SINGLE_PRECISION build (WP=4) each such transfer reads/writes twice the buffer - heap corruption on the broadcasts, garbage physics in the forcing normalization. Use MPI_WP and real(kind=WP) throughout, as main does since the single-precision support landed; double-precision builds are unchanged. The fbin gather-to-master routines keep real64, matching the convention of the neighbouring gather routines.
…tep reads apply_idemix2_restart moved the restart-saved divergence from slot 1 to slot 2 and zeroed slot 1. But the first step after restart swaps to ti=2/tip1=1: the fresh divergence is computed into slot 2 (clobbering the saved values unread) and the AB2 'previous divergence' is read from slot 1 (just zeroed). The saved divh/divs restart fields were therefore never used and the first post-restart step degraded to an over-weighted Euler step, making restarts non-reproducible against a continuous run. Keep the saved divergence in slot 1 and zero slot 2 instead; the E-field staging was already correct.
cvmix_IDEMIX2 took digit 7 and cvmix_TIDAL moved to digit 8, but only cvmix_PP+cvmix_TIDAL (47->48) and the oce_ale branch checks were renumbered. KPP+cvmix_TIDAL, PP+cvmix_TIDAL and cvmix_KPP+cvmix_TIDAL still mapped to 17/27/37: their base scheme never ran (oce_ale checks 18/28/38) and mod(...,10)==7 routed them into IDEMIX2 instead of TIDAL. Move the three mappings to 18/28/38 and update every consumer (oce_setup_step, oce_ale_tracer, gen_modules_diag, io_meandata, oce_ale_pressure_bv) to match.
…e-path ORDERED region The spectral flux-to-tracer edge loop emits an !$OMP ORDERED region on the __openmp_reproducible path but its !$OMP DO lacked the required ORDERED clause - the same defect class PR #978 eliminates tree-wide, and its ctest guard flags this file the moment both merge. Guard the clause so the default OpenMP (lock-based) build is byte-identical. Verified with PR #978's check_omp_ordered.py: this file now passes both the reproducible and default invariants.
areasvol is zero above ulevels_nod2D: for a node under an ice-shelf cavity (ulevels>1) areasvol(1,node) is exactly 0, so the element-to-node averages in interp_e2n_1d/2d/3d divide by zero the moment IDEMIX2 runs on a cavity mesh (Inf/NaN, or an FPE abort). Use the area of the node's first wet level; identical on non-cavity meshes where ulevels==1.
…ision WP
The module hard-coded its kind to the external library's cvmix_r8
(double). In a USE_SINGLE_PRECISION build the driver's real(WP)=real32
fields then match no specific of the generic interfaces and the build
fails ('There is no specific subroutine for the generic
cvmix_idemix2_compute_...', 19 call sites) - exposed by sp_ctests once
main was merged. Unlike the external CVMix library (kept double behind a
conversion boundary, cf. gen_modules_cvmix_tke/idemix), this module is
FESOM-owned: bind its kind to WP (keeping the cvmix_r8 name to avoid
touching every declaration) so all driver calls match in both builds.
The single call into the fixed real*8 library helper solve_tridiag now
converts at its call site; double-precision builds are bit-identical
throughout.
|
@patrickscholz: Adapted to current state of main branch. Mostly regarding single precision, and some omp ordering. Please take a look. |
IDEMIX2: Internal Wave Energy Parameterisation for FESOM2
IDEMIX2 (Internal wave Dissipation, Energy and MIXing, version 2) is a prognostic parameterisation of internal gravity wave energy and its contribution to diapycnal mixing, based on Olbers & Eden (2013) and Eden et al. (2014). It is implemented here as an extension to the existing IDEMIX1/TKE mixing scheme in FESOM2.
Concept
Unlike IDEMIX1, which tracks a single vertically integrated internal wave energy field, IDEMIX2 resolves the horizontal propagation direction of internal waves through a spectral angular dimension (
nfbindirectional bins, default 50 interior + 2 ghost bins covering 0–2π). This captures anisotropic wave fields near generation sites such as rough topography and coastlines, and allows wave energy to be tracked as it propagates across ocean basins.Wave energy compartments
Two spectral wave energy fields are evolved independently:
Timestep structure
Each call to
calc_cvmix_idemix2follows this sequence:Reflective coastal boundary condition
Wave energy reaching a coastline is redirected into the spectrally mirrored propagation direction at the last interior node, rather than leaking into land:
The mirror bin
φ_kfor each coast edge is pre-computed once at init from the edge geometry and stored iniwe2_refl_bin(edge).Forcing inputs
E_M2surface sourceE_niwsurface source orE_iwsurface BCE_iwbottom source