Fix uninitialised GM streamfunction at degenerate cavity level ranges, and two out-of-bounds reads - #960
Open
JanStreffing wants to merge 1 commit into
Open
Fix uninitialised GM streamfunction at degenerate cavity level ranges, and two out-of-bounds reads#960JanStreffing wants to merge 1 commit into
JanStreffing wants to merge 1 commit into
Conversation
fer_solve_Gamma solves the GM streamfunction over nzmin = ulevels_nod2D_max(n), nzmax = nlevels_nod2D_min(n) (max/min over the node's surrounding elements). When a node touches both a deep cavity element and a shallow one these can invert (nzmin >= nzmax). Both sweeps then run zero times, but "tr(:,nzmax) = tp(:,nzmax)" still executes and publishes an unassigned element of the automatic array tp into fer_gamma. Such a node carries no GM transport, so zero its column and skip. Also guard two non-short-circuit .and. expressions that read index 0: ice_maEVP.F90 ulevels(edge_tri(2,ed)) at boundary edges (edge_tri(2,ed)==0) io_xios.F90 angles(k) at the end of each insertion-sort sweep
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three independent fixes found while tracking down a reproducible leg-start blow-up in a coupled AWI-ESM3 run with a moving (PISM-driven) ice-shelf cavity. The first is the crash; the other two are latent out-of-bounds reads found with a bounds-checked build along the way.
1.
fer_solve_Gamma: unassigned automatic-array element published intofer_gammaThe bug
fer_solve_Gamma(src/oce_fer_gm.F90) solves the GM streamfunction overA node that touches both a deep cavity element and a shallow one can end up with
nzmin >= nzmax, i.e. no interior range to solve on. Both sweeps then execute zero times:but
tr(:,nzmax) = tp(:,nzmax)is unconditional, so an unassigned element of the local automatic arraytpis written straight intofer_gamma.Why it is destructive
fer_gamma2velturns that into a bolus velocity,and
solve_tracers_aleaddsfer_uvtoUV, advects the tracers with it, then subtracts it back.Measured in the failing run (one bad gamma at the level-5 interface, node shared by three elements):
The antisymmetric pair is the signature of a single corrupted
fer_gammaat that interface;helemand the meridional component are healthy. Consequences in one timestep:tempreaches 1e250,saltpins at its 3/45 bounds;0.0inuwherever|fer_uv| >> |UV|, because(x + y) - yroundsxaway entirely;Wvel/Wvel_eget the same treatment viafer_Wvel, so the vertical velocity acquires exact-zero holes too.Why it is intermittent
tp(:,nzmax)is whatever the stack holds, so the outcome depends on memory layout. In our case the same source crashed atmstep 1under-O3and ran fine under-check bounds -check pointers -check uninit -g. Degenerate nodes were present in most of our meshes (0–2 per mesh) without being fatal, which is why this can sit latent for a long time.-check boundsdoes not catch it:fer_gammais written in bounds; the defect is a read of an unassigned value.Fix
A node with no interior range carries no GM transport, so zero its column and skip:
Requires
Fer_GM = .true.; the degenerate range requiresuse_cavity = .true..2 & 3. Two non-short-circuit
.and.expressions reading index 0Fortran does not guarantee short-circuit evaluation of
.and., so the compiler may evaluate the right operand even when the left is false.src/ice_maEVP.F90(bothEVPdynamics_mandEVPdynamics_a), applying the sea-ice velocity boundary condition at the cavity–ocean edge:At a boundary edge
edge_tri(2,ed) == 0, soulevels(0)is read. The value read then decides whether the ice velocity at that edge is zeroed, i.e. a physics branch taken on out-of-bounds memory. Only reachable withuse_cavity = .true..src/io_xios.F90, insertion sort of cell-corner angles:reads
angles(0)at the end of every sweep. Harmless in practice, but it aborts any bounds-checked build during initialisation.Both are rewritten so the guard is evaluated separately. No behavioural change other than removing the out-of-bounds access.
Testing
mstep 1on every attempt, bit-identically; with fix 1 it ran past the failure point (day 27, step 1920) with no blow-up.main(uncoupled Release build, Intel).