Skip to content

Fix uninitialised GM streamfunction at degenerate cavity level ranges, and two out-of-bounds reads - #960

Open
JanStreffing wants to merge 1 commit into
mainfrom
fix/cavity-gm-degenerate-level-range
Open

Fix uninitialised GM streamfunction at degenerate cavity level ranges, and two out-of-bounds reads#960
JanStreffing wants to merge 1 commit into
mainfrom
fix/cavity-gm-degenerate-level-range

Conversation

@JanStreffing

Copy link
Copy Markdown
Collaborator

Three independent fixes found while tracking down a reproducible leg-start blow-up in a coupled AWI-ESM3 run with a moving (PISM-driven) ice-shelf cavity. The first is the crash; the other two are latent out-of-bounds reads found with a bounds-checked build along the way.

1. fer_solve_Gamma: unassigned automatic-array element published into fer_gamma

The bug

fer_solve_Gamma (src/oce_fer_gm.F90) solves the GM streamfunction over

nzmax = nlevels_nod2D_min(n)   ! min over the node's surrounding elements
nzmin = ulevels_nod2D_max(n)   ! max over the node's surrounding elements

A node that touches both a deep cavity element and a shallow one can end up with nzmin >= nzmax, i.e. no interior range to solve on. Both sweeps then execute zero times:

DO nz = nzmin+1, nzmax          ! e.g. 8..5 -> zero iterations
   ...
END DO
tr(:,nzmax) = tp(:,nzmax)       ! <-- still runs; tp(:,nzmax) never assigned
do nz = nzmax-1, nzmin, -1      ! e.g. 4..7 step -1 -> zero iterations

but tr(:,nzmax) = tp(:,nzmax) is unconditional, so an unassigned element of the local automatic array tp is written straight into fer_gamma.

Why it is destructive

fer_gamma2vel turns that into a bolus velocity,

fer_uv(1,nz,el) = sum(fer_gamma(1,nz,elnod) - fer_gamma(1,nz+1,elnod))/(3*helem(nz,el))

and solve_tracers_ale adds fer_uv to UV, advects the tracers with it, then subtracts it back.

Measured in the failing run (one bad gamma at the level-5 interface, node shared by three elements):

fer_uv(1, nz=4) = -1.63611672e+251     fer_uv(2, nz=4) = +1.8e-04    helem = 10.0
fer_uv(1, nz=5) = +1.63611672e+251     fer_uv(2, nz=5) = +3.6e-06    helem = 10.0

The antisymmetric pair is the signature of a single corrupted fer_gamma at that interface; helem and the meridional component are healthy. Consequences in one timestep:

  • tracers advected at ~1e251 m/s: temp reaches 1e250, salt pins at its 3/45 bounds;
  • the subtract-back leaves exactly 0.0 in u wherever |fer_uv| >> |UV|, because (x + y) - y rounds x away entirely;
  • Wvel/Wvel_e get the same treatment via fer_Wvel, so the vertical velocity acquires exact-zero holes too.

Why it is intermittent

tp(:,nzmax) is whatever the stack holds, so the outcome depends on memory layout. In our case the same source crashed at mstep 1 under -O3 and ran fine under -check bounds -check pointers -check uninit -g. Degenerate nodes were present in most of our meshes (0–2 per mesh) without being fatal, which is why this can sit latent for a long time.

-check bounds does not catch it: fer_gamma is written in bounds; the defect is a read of an unassigned value.

Fix

A node with no interior range carries no GM transport, so zero its column and skip:

if (nzmin >= nzmax) then
    tr(:,:) = 0.0_WP
    cycle
end if

Requires Fer_GM = .true.; the degenerate range requires use_cavity = .true..

2 & 3. Two non-short-circuit .and. expressions reading index 0

Fortran does not guarantee short-circuit evaluation of .and., so the compiler may evaluate the right operand even when the left is false.

src/ice_maEVP.F90 (both EVPdynamics_m and EVPdynamics_a), applying the sea-ice velocity boundary condition at the cavity–ocean edge:

if ( (ulevels(edge_tri(1,ed))>1) .or. &
    ( edge_tri(2,ed)>0 .and. ulevels(edge_tri(2,ed))>1) ) then

At a boundary edge edge_tri(2,ed) == 0, so ulevels(0) is read. The value read then decides whether the ice velocity at that edge is zeroed, i.e. a physics branch taken on out-of-bounds memory. Only reachable with use_cavity = .true..

src/io_xios.F90, insertion sort of cell-corner angles:

do while (k >= 1 .and. angles(k) > tmp_ang)

reads angles(0) at the end of every sweep. Harmless in practice, but it aborts any bounds-checked build during initialisation.

Both are rewritten so the guard is evaluated separately. No behavioural change other than removing the out-of-bounds access.

Testing

  • The failing leg previously blew up at mstep 1 on every attempt, bit-identically; with fix 1 it ran past the failure point (day 27, step 1920) with no blow-up.
  • Fixes 2 and 3 were verified against a bounds-checked build, which traps on them without the change and passes with it.
  • Compile-checked against main (uncoupled Release build, Intel).

fer_solve_Gamma solves the GM streamfunction over
  nzmin = ulevels_nod2D_max(n), nzmax = nlevels_nod2D_min(n)
(max/min over the node's surrounding elements). When a node touches both a
deep cavity element and a shallow one these can invert (nzmin >= nzmax). Both
sweeps then run zero times, but "tr(:,nzmax) = tp(:,nzmax)" still executes and
publishes an unassigned element of the automatic array tp into fer_gamma.
Such a node carries no GM transport, so zero its column and skip.

Also guard two non-short-circuit .and. expressions that read index 0:
  ice_maEVP.F90  ulevels(edge_tri(2,ed)) at boundary edges (edge_tri(2,ed)==0)
  io_xios.F90    angles(k) at the end of each insertion-sort sweep
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant