From c720eb1114c4fb13ab64919e3998bb38c6d8e226 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mike=20Gro=C3=9Fmann?= Date: Thu, 23 Jul 2026 15:09:26 +0200 Subject: [PATCH] tests: assert normal address-space termination in shutdown acceptance The shutdown acceptance test asserted only on the in-flight crash signatures (S33E, repeated __CRTGET spam, SVC-dump markers). After the 2026-07-23 libc370 recovery-exit relink those are gone, yet the address space still ends abnormally: an mvsMF poll worker that ignores quiesce leaves a live subtask, and P HTTPD drives IEF450I ... ABEND SA03. The old assertions pass that run vacuously. Add the missing address-space-end assertions: - FAIL on IEF450I ... ABEND and $HASP310 ... TERMINATED AT END OF MEMORY. SVC-dump (IEA911E) was already covered by the svcdump group; note it there. - Confirm the address-space end was captured in the window ($HASP395 ... ENDED or IEF404I, scoped to JOBNAME) before reporting PASS, so a truncated log is not mistaken for a clean shutdown. Those two records appear on clean and abnormal ends alike, so they gate log-completeness, not health. - Add JOBNAME (default HTTPD) to scope the end-capture markers. - Document HTTPD002I/HTTPD060I worker-shutdown WTOs as diagnostic only, with a CAUTION when a PASS shows none. The __CRTGET assertion is unchanged (zero-tolerance): correct for the termination path this test drives. Refs #122 --- tests/README.md | 32 +++++++---- tests/shutdown-acceptance.sh | 103 ++++++++++++++++++++++++++++++----- 2 files changed, 111 insertions(+), 24 deletions(-) diff --git a/tests/README.md b/tests/README.md index c957a1a..41dc649 100644 --- a/tests/README.md +++ b/tests/README.md @@ -10,7 +10,11 @@ Acceptance criterion for [#122](https://github.com/mvslovers/httpd/issues/122) (epic [mvsmf#177](https://github.com/mvslovers/mvsmf/issues/177), Workstream D): after a request handler faults, `P HTTPD` must terminate the address space cleanly — **no `S33E`**, **no repeated `__CRTGET CRT for TCB(...) was not found -in PPA(...)`**, **no SVC dump**. +in PPA(...)`**, **no SVC dump**, and the address space must **end normally** +(no `IEF450I ... ABEND`, no `$HASP310 ... TERMINATED AT END OF MEMORY`). The +`$HASP395 ... ENDED` / `IEF404I` end records are used only to confirm the end +was *captured* in the window — they appear on clean and abnormal ends alike and +do not themselves prove health. The code fix lives in libc370 (worker DETACH gating + recovery-exit hardening). This test observes the symptom in httpd's own address space. @@ -28,15 +32,22 @@ This test observes the symptom in httpd's own address space. ### Result -- **FAIL** — a fault marker is present *and* any of `S33E` / `__CRTGET` spam / - SVC dump appears. This is the #122 bug reproduced and is the **expected - result on the current build**. -- **PASS** — a handler faulted but shutdown was clean. The post-fix target. - A PASS on a build that still carries the libc370 defect is suspicious — the - banner says so. -- **INCONCLUSIVE** — no fault marker (`HTTPD062E` / `MVSMF99E`) in the window, - so no handler actually faulted. The test did not exercise #122; fix the - provocation and re-run. +- **FAIL** — a fault marker is present *and* either a crash signature (`S33E` / + `__CRTGET` spam / SVC dump) **or** an abnormal address-space end (`IEF450I ... + ABEND`, `$HASP310 ... TERMINATED AT END OF MEMORY`) appears in the window. + After the 2026-07-23 recovery-exit relink the residual symptom is the abnormal + end (`SA03`), so this class matters even when no crash spam appears. +- **PASS** — a handler faulted, no crash signature and no abnormal end appeared + (`IEF450I` / `$HASP310` / SVC dump all absent), and the address-space end was + captured in the window (`$HASP395 ... ENDED` or `IEF404I`). A PASS on a build + that still carries the libc370 defect is suspicious — the banner says so. + `HTTPD002I` / `HTTPD060I` worker-shutdown WTOs corroborate a clean drain but do + **not** gate the result; a PASS with none of them present emits a CAUTION. +- **INCONCLUSIVE** — either no fault marker (`HTTPD062E` / `MVSMF99E`) in the + window (no handler faulted), or no crash/abnormal signature yet the + address-space end was not captured (`$HASP395 ... ENDED` / `IEF404I` both + absent — a truncated log looks the same as a clean end). Fix the provocation + or extend the capture and re-run. Exit codes: `0` pass, `1` fail, `2` config error, `3` inconclusive. @@ -90,6 +101,7 @@ Environment variables (or a sourced `.env`): | `FAULT_MARK` | `HTTPD062E\|MVSMF99E` | proof-of-fault regex | | `HTTPD_AUTH` | (unset) | `user:pass` if `LOGIN` gates GET | | `WAIT_SECS` | `120` | max wait for the AS to end | +| `JOBNAME` | `HTTPD` | MVS jobname of the HTTPD STC (scopes the end-capture assertion) | | `STOP_ADAPTER` | `manual` | `manual` or `cmd` | | `STOP_CMD` | (unset) | command to issue `P HTTPD` when `cmd` | | `CONSOLE_LOG` | (required) | captured console/hardcopy log to assert over | diff --git a/tests/shutdown-acceptance.sh b/tests/shutdown-acceptance.sh index ff8f037..5d2d0bd 100755 --- a/tests/shutdown-acceptance.sh +++ b/tests/shutdown-acceptance.sh @@ -3,7 +3,11 @@ # # Verifies that stopping HTTPD (P HTTPD) AFTER a request handler has faulted # terminates the address space cleanly — no S33E, no repeated "__CRTGET CRT -# for TCB(...) was not found in PPA(...)", no SVC dump. +# for TCB(...) was not found in PPA(...)", no SVC dump, and a NORMAL address- +# space end: no IEF450I ... ABEND, no "$HASP310 ... TERMINATED AT END OF +# MEMORY", no SVC-dump (IEA911E). The "$HASP395 ... ENDED" / IEF404I end +# records only confirm the end was CAPTURED in the window — they appear on +# clean and abnormal ends alike and do not themselves prove health. # # Background: on shutdown, worker teardown DETACHes tasks without proving they # terminated (S33E), and the worker recovery ESTAE runs the C runtime under a @@ -16,7 +20,7 @@ # worker in the post-fault state #122 needs. See the result banner at the end. # # --------------------------------------------------------------------------- -# TWO THINGS THAT SILENTLY BREAK THIS TEST — read before editing: +# THINGS THAT SILENTLY BREAK THIS TEST — read before editing: # # 1. WINDOWED ASSERTION. CONSOLE_LOG is typically an append-only Hercules # hardcopy log that ALREADY contains S33E / __CRTGET lines from earlier @@ -31,6 +35,25 @@ # caught by mvsMF's inner ESTAE instead and logs MVSMF99E, so the worker # try() returns 0 and HTTPD062E never fires. Accepting either keeps the # test valid whether ABEND_PATH is core (/abend) or a CGI endpoint. +# +# 3. ABNORMAL ADDRESS-SPACE END. A clean in-flight run (no S33E/__CRTGET/SVC +# dump) is NOT enough. After the 2026-07-23 recovery-exit relink the +# symptom moved from S33E to an abnormal address-space end (IEF450I ... +# ABEND SA03). This test therefore also fails on "IEF450I ... ABEND", +# "$HASP310 ... TERMINATED AT END OF MEMORY", and SVC-dump markers (IEA911E, +# via the svcdump group). The "$HASP395 ... ENDED" / IEF404I records do NOT +# discriminate health: they appear on clean AND abnormal ends alike (S33E, +# SA03, EOM). They are used only to confirm the end was CAPTURED in the +# window (end_seen), so a truncated log is not mistaken for a clean shutdown. +# The failure patterns are kept LOOSE (not scoped to $JOBNAME) so a stray +# match only over-reports FAIL; the end-capture markers are scoped TIGHT to +# $JOBNAME so a miss downgrades to INCONCLUSIVE. Both err toward safe. +# +# 4. HTTPD060I / HTTPD002I ARE DIAGNOSTIC, NOT A GATE. Worker- and server- +# shutdown WTOs corroborate an orderly drain but never decide the verdict: +# the authoritative signal is the address-space end (note 3). The SA03 run +# produced zero HTTPD060I, so a PASS with zero of them emits a CAUTION, not +# a FAIL. Do not turn these into a hard requirement. # --------------------------------------------------------------------------- # Configuration (override via environment or a sourced .env) # --------------------------------------------------------------------------- @@ -41,6 +64,7 @@ # FAULT_MARK regex proving a handler faulted (default: HTTPD062E|MVSMF99E) # HTTPD_AUTH optional "user:pass" if LOGIN gates GET (default: unset) # WAIT_SECS max seconds to wait for the AS to end (default: 120) +# JOBNAME MVS jobname of the HTTPD started task (default: HTTPD) # # STOP_ADAPTER how "P HTTPD" is issued: manual | cmd (default: manual) # STOP_CMD command line to issue P HTTPD when STOP_ADAPTER=cmd @@ -68,6 +92,7 @@ ABEND_HITS=${ABEND_HITS:-12} FAULT_MARK=${FAULT_MARK:-HTTPD062E|MVSMF99E} WAIT_SECS=${WAIT_SECS:-120} STOP_ADAPTER=${STOP_ADAPTER:-manual} +JOBNAME=${JOBNAME:-HTTPD} fail() { printf '%s\n' "$*" >&2; exit 2; } @@ -148,21 +173,43 @@ tail -c +"$((log_offset + 1))" "$CONSOLE_LOG" > "$window" 2>/dev/null || cp "$CO # proof the provocation actually faulted a worker — HTTPD062E (core) or MVSMF99E (CGI) faultmarks=$(grep -Ec "$FAULT_MARK" "$window" 2>/dev/null || true) -# failure signatures from #122 +# in-flight crash signatures from #122 s33e=$(grep -c "S33E" "$window" 2>/dev/null || true) crtget=$(grep -Ec "__CRTGET CRT for TCB.*was not found in PPA" "$window" 2>/dev/null || true) svcdump=$(grep -Eic "SVC dump|SDUMP|IEA911E|IEA794I" "$window" 2>/dev/null || true) -# clean-shutdown positive markers (httpd.c:335 / :701) +# abnormal address-space END — the residual after the 2026-07-23 relink (S33E -> SA03). +# The svcdump group above (IEA911E/...) is the third abnormal-end discriminator. +# LOOSE (not scoped to $JOBNAME): a stray match only over-reports FAIL, the safe way. Note 3. +abend=$(grep -Ec "IEF450I.*ABEND" "$window" 2>/dev/null || true) +hasp310=$(grep -Ec "[$]HASP310.*TERMINATED AT END OF MEMORY" "$window" 2>/dev/null || true) +# address-space END records — "$HASP395 ... ENDED" / IEF404I, both for $JOBNAME. These are +# WINDOW-COMPLETENESS evidence only (see end_seen), NOT health signals — both appear on +# clean AND abnormal ends. TIGHT to $JOBNAME so a mismatch downgrades to INCONCLUSIVE, +# not a false PASS. Note 3. +hasp395=$(grep -Ec "[$]HASP395[[:space:]]+${JOBNAME}[[:space:]].*ENDED" "$window" 2>/dev/null || true) +ief404=$(grep -Ec "IEF404I[[:space:]]+${JOBNAME}[[:space:]].*ENDED" "$window" 2>/dev/null || true) +# clean-shutdown WTOs (httpd.c:335 / :701) — DIAGNOSTIC ONLY, never a gate. Note 4. clean=$(grep -Ec "HTTPD002I Server is SHUTDOWN|HTTPD060I SHUTDOWN worker" "$window" 2>/dev/null || true) : "${faultmarks:=0}" "${s33e:=0}" "${crtget:=0}" "${svcdump:=0}" "${clean:=0}" +: "${abend:=0}" "${hasp310:=0}" "${hasp395:=0}" "${ief404:=0}" + +# end_seen = the address-space END was CAPTURED in the window (NOT that it ended +# normally). "$HASP395 ... ENDED" and IEF404I both appear on clean AND abnormal ends, +# so they cannot judge health — IEF450I / $HASP310 / SVC dump do that. OR because either +# record alone proves the end was captured; requiring both would falsely flag a healthy +# run when JES config omits one. +if [ "$hasp395" -ne 0 ] || [ "$ief404" -ne 0 ]; then end_seen=1; else end_seen=0; fi echo "-- assertion over captured console window --" echo " fault markers ($FAULT_MARK) : $faultmarks" echo " S33E : $s33e" echo " __CRTGET ... not found in PPA : $crtget" echo " SVC dump markers : $svcdump" -echo " clean-shutdown markers : $clean" +echo " IEF450I ... ABEND (abnormal AS end) : $abend" +echo " \$HASP310 TERMINATED AT END OF MEMORY : $hasp310" +echo " AS end captured (\$HASP395/IEF404I) : $end_seen" +echo " clean-shutdown WTOs (diagnostic) : $clean" echo rm -f "$window" @@ -179,24 +226,52 @@ EOF exit 3 fi -if [ "$s33e" -ne 0 ] || [ "$crtget" -ne 0 ] || [ "$svcdump" -ne 0 ]; then +if [ "$s33e" -ne 0 ] || [ "$crtget" -ne 0 ] || [ "$svcdump" -ne 0 ] \ + || [ "$abend" -ne 0 ] || [ "$hasp310" -ne 0 ]; then cat <