diff --git a/tests/README.md b/tests/README.md index 41dc649..2a94a54 100644 --- a/tests/README.md +++ b/tests/README.md @@ -23,13 +23,51 @@ This test observes the symptom in httpd's own address space. 1. Records the current byte offset of `CONSOLE_LOG` (the assertion window starts here — see "Windowed assertion" below). -2. Provokes a faulting handler: `ABEND_HITS` requests to `ABEND_PATH` - (default `/abend`, whose handler executes `DC H'0'` → S0C1 inside the - worker request pipeline). Repeated hits poison the whole worker pool. +2. Provokes the pre-stop state under test (`PROVOKE`, see "Provocation modes"): + either faulting handlers (`abend`) or workers parked in an in-flight long + poll (`longpoll`, the `#179` / `SA03` path). 3. Issues `P HTTPD` (see adapters below). 4. Waits up to `WAIT_SECS` for the listener to stop accepting. 5. Asserts over the console-log slice appended during this run. +### Provocation modes (`PROVOKE`) + +Which shutdown path the run exercises: + +- **`abend`** (default) — `ABEND_HITS` requests to `ABEND_PATH` (default + `/abend`, whose handler executes `DC H'0'` → S0C1). Handlers fault *fast*, so + this drives libc370's **recovery-drain** path. Proof-of-scenario is a fault + marker (`HTTPD062E` / `MVSMF99E`) in the window. +- **`longpoll`** — launches `LONGPOLL_N` (default 4) concurrent, blocking + long-poll requests (`LONGPOLL_CMD`) and checks that at least one is *still in + flight* `LONGPOLL_SETTLE` s later, then issues `P HTTPD` while they are parked. + This is the only mode that reproduces `#179` / the `SA03`: a worker parked in a + long poll cannot drain inside libc370's ~5 s window. Proof-of-scenario is the + in-flight count — **not** a fault marker (a clean parked poll faults nothing). + +> A green **`abend`** run does **not** establish the `#179` fix — it tests a +> different path. Use **`longpoll`** to gate `#179` (mvslovers/mvsmf#179). + +Example `LONGPOLL_CMD` — one synchronous unsolicited-message detection PUT that +blocks up to 60 s (mvsMF console **issue-command** endpoint, +`PUT /zosmf/restconsoles/consoles/{console-name}` — *not* the `/solmsgs` collect +sub-resource): + +```sh +LONGPOLL_CMD='curl -s -o /dev/null -u USER:PASS -X PUT \ + -H "Content-Type: application/json" \ + -d "{\"cmd\":\"D T\",\"unsol-key\":\"IEE136I\",\"unsol-detect-sync\":\"Y\",\"unsol-detect-timeout\":\"60\"}" \ + http://HOST:PORT/zosmf/restconsoles/consoles/defcn' +``` + +The long poll must **still be running when `P HTTPD` is issued** — that is the +whole point. The in-flight count is measured `LONGPOLL_SETTLE` s after launch, +but the stop lands later. In `STOP_ADAPTER=cmd` mode the stop follows +immediately, so a `unsol-detect-timeout` of ~60 s is ample. In `manual` mode the +operator's reaction time is unbounded: set `unsol-detect-timeout` comfortably +above how long you expect to take typing `P HTTPD`, or the polls will have +already returned (no worker parked → INCONCLUSIVE, not a real reproduction). + ### Result - **FAIL** — a fault marker is present *and* either a crash signature (`S33E` / @@ -43,11 +81,12 @@ This test observes the symptom in httpd's own address space. that still carries the libc370 defect is suspicious — the banner says so. `HTTPD002I` / `HTTPD060I` worker-shutdown WTOs corroborate a clean drain but do **not** gate the result; a PASS with none of them present emits a CAUTION. -- **INCONCLUSIVE** — either no fault marker (`HTTPD062E` / `MVSMF99E`) in the - window (no handler faulted), or no crash/abnormal signature yet the - address-space end was not captured (`$HASP395 ... ENDED` / `IEF404I` both - absent — a truncated log looks the same as a clean end). Fix the provocation - or extend the capture and re-run. +- **INCONCLUSIVE** — either the intended scenario didn't run (in `abend`, no + fault marker `HTTPD062E` / `MVSMF99E`; in `longpoll`, no long-poll still in + flight at `P HTTPD`), or no crash/abnormal signature yet the address-space end + was not captured (`$HASP395 ... ENDED` / `IEF404I` both absent — a truncated + log looks the same as a clean end). Fix the provocation or extend the capture + and re-run. Exit codes: `0` pass, `1` fail, `2` config error, `3` inconclusive. @@ -102,6 +141,10 @@ Environment variables (or a sourced `.env`): | `HTTPD_AUTH` | (unset) | `user:pass` if `LOGIN` gates GET | | `WAIT_SECS` | `120` | max wait for the AS to end | | `JOBNAME` | `HTTPD` | MVS jobname of the HTTPD STC (scopes the end-capture assertion) | +| `PROVOKE` | `abend` | pre-stop scenario: `abend` (fault handlers) or `longpoll` (park workers) | +| `LONGPOLL_CMD` | (required for `longpoll`) | one blocking long-poll request, run `LONGPOLL_N`× concurrently | +| `LONGPOLL_N` | `4` | `longpoll`: concurrent in-flight long-polls | +| `LONGPOLL_SETTLE` | `3` | `longpoll`: seconds to let them park before `P HTTPD` | | `STOP_ADAPTER` | `manual` | `manual` or `cmd` | | `STOP_CMD` | (unset) | command to issue `P HTTPD` when `cmd` | | `CONSOLE_LOG` | (required) | captured console/hardcopy log to assert over | diff --git a/tests/shutdown-acceptance.sh b/tests/shutdown-acceptance.sh index 5d2d0bd..26f31d5 100755 --- a/tests/shutdown-acceptance.sh +++ b/tests/shutdown-acceptance.sh @@ -54,6 +54,16 @@ # the authoritative signal is the address-space end (note 3). The SA03 run # produced zero HTTPD060I, so a PASS with zero of them emits a CAUTION, not # a FAIL. Do not turn these into a hard requirement. +# +# 5. PROVOCATION MODE decides WHICH shutdown path you test. Default +# PROVOKE=abend faults handlers -- that is the recovery-drain path +# (libc370#9), NOT a worker parked in a long poll. The #179 / SA03 crash +# needs a worker still IN FLIGHT in a long poll when P HTTPD lands, which a +# fast fault cannot create; PROVOKE=longpoll launches $LONGPOLL_N concurrent +# blocking long-polls (LONGPOLL_CMD, e.g. an unsol-detect-sync console PUT) +# and proves >=1 is still parked at stop. Proof-of-scenario is that in-flight +# count, NOT a fault marker -- a clean parked poll faults nothing. A green +# abend run does NOT establish the #179 fix; use longpoll for that. # --------------------------------------------------------------------------- # Configuration (override via environment or a sourced .env) # --------------------------------------------------------------------------- @@ -66,6 +76,16 @@ # WAIT_SECS max seconds to wait for the AS to end (default: 120) # JOBNAME MVS jobname of the HTTPD started task (default: HTTPD) # +# PROVOKE pre-stop scenario: abend | longpoll (default: abend) +# abend -> fault handlers (recovery-drain path; libc370#9). +# longpoll -> park workers in an in-flight long poll at P HTTPD +# (the #179 / SA03 path a fast fault cannot create). +# LONGPOLL_CMD PROVOKE=longpoll, REQUIRED: one blocking long-poll request +# (e.g. an unsol-detect-sync console PUT). Run $LONGPOLL_N times, +# concurrently, in the background — see tests/README.md. +# LONGPOLL_N PROVOKE=longpoll: concurrent long-polls (default: 4) +# LONGPOLL_SETTLE PROVOKE=longpoll: seconds to let them park pre-stop (default: 3) +# # STOP_ADAPTER how "P HTTPD" is issued: manual | cmd (default: manual) # STOP_CMD command line to issue P HTTPD when STOP_ADAPTER=cmd # (e.g. a Hercules console pipe, or a one-shot operator-command @@ -93,6 +113,9 @@ FAULT_MARK=${FAULT_MARK:-HTTPD062E|MVSMF99E} WAIT_SECS=${WAIT_SECS:-120} STOP_ADAPTER=${STOP_ADAPTER:-manual} JOBNAME=${JOBNAME:-HTTPD} +PROVOKE=${PROVOKE:-abend} +LONGPOLL_N=${LONGPOLL_N:-4} +LONGPOLL_SETTLE=${LONGPOLL_SETTLE:-3} fail() { printf '%s\n' "$*" >&2; exit 2; } @@ -104,10 +127,15 @@ CURL="curl -s -S -o /dev/null -w %{http_code} --max-time 10" BASE="http://${HTTPD_HOST}:${HTTPD_PORT}" echo "== httpd#122 shutdown acceptance ==" -echo " target : $BASE" -echo " abend : $ABEND_PATH x $ABEND_HITS" -echo " stop : $STOP_ADAPTER" -echo " log : $CONSOLE_LOG" +echo " target : $BASE" +echo " provoke : $PROVOKE" +if [ "$PROVOKE" = abend ]; then + echo " abend : $ABEND_PATH x $ABEND_HITS" +else + echo " longpoll: $LONGPOLL_N concurrent, settle ${LONGPOLL_SETTLE}s" +fi +echo " stop : $STOP_ADAPTER" +echo " log : $CONSOLE_LOG" echo # --- 1. mark the log window START (byte offset) — see note #1 above --------- @@ -118,21 +146,48 @@ log_offset=$(printf '%s' "$log_offset" | tr -d ' ') echo "-- console-log window starts at byte offset $log_offset --" echo -# --- 2. provoke: fault the handler enough times to poison the worker pool --- -echo "-- provoking handler abend ($ABEND_PATH) --" -faulted=0 -i=0 -while [ "$i" -lt "$ABEND_HITS" ]; do - i=$((i + 1)) - code=$($CURL "${BASE}${ABEND_PATH}" 2>/dev/null) || code="conn-reset" - # A faulting worker drops the connection or returns 5xx; a clean 200/404 - # means the handler did NOT fault (wrong path / already handled otherwise). - case "$code" in - 5*|conn-reset|000) faulted=$((faulted + 1)) ;; - esac - printf ' hit %2d -> %s\n' "$i" "$code" -done -echo " handler-fault responses: $faulted / $ABEND_HITS" +# --- 2. provoke: put the worker pool into the pre-stop state under test ----- +# abend -> handlers fault (fast; the recovery-drain path, libc370#9). +# longpoll -> workers PARKED in an in-flight long poll still running when P HTTPD +# lands — the #179 / SA03 path a fast fault cannot reproduce. +LONGPOLL_PIDS="" +inflight=0 +case "$PROVOKE" in +abend) + echo "-- provoking handler abend ($ABEND_PATH x $ABEND_HITS) --" + faulted=0 + i=0 + while [ "$i" -lt "$ABEND_HITS" ]; do + i=$((i + 1)) + code=$($CURL "${BASE}${ABEND_PATH}" 2>/dev/null) || code="conn-reset" + # A faulting worker drops the connection or returns 5xx; a clean 200/404 + # means the handler did NOT fault (wrong path / already handled otherwise). + case "$code" in + 5*|conn-reset|000) faulted=$((faulted + 1)) ;; + esac + printf ' hit %2d -> %s\n' "$i" "$code" + done + echo " handler-fault responses: $faulted / $ABEND_HITS" + ;; +longpoll) + [ -n "${LONGPOLL_CMD:-}" ] || fail "PROVOKE=longpoll requires LONGPOLL_CMD (one blocking long-poll request)." + echo "-- launching $LONGPOLL_N concurrent long-polls (must be in flight at P HTTPD) --" + trap '[ -n "$LONGPOLL_PIDS" ] && kill $LONGPOLL_PIDS 2>/dev/null' EXIT INT TERM + i=0 + while [ "$i" -lt "$LONGPOLL_N" ]; do + i=$((i + 1)) + sh -c "$LONGPOLL_CMD" >/dev/null 2>&1 & + LONGPOLL_PIDS="$LONGPOLL_PIDS $!" + done + # let the requests reach and PARK in the poll loop before we stop the server + sleep "$LONGPOLL_SETTLE" + for p in $LONGPOLL_PIDS; do + kill -0 "$p" 2>/dev/null && inflight=$((inflight + 1)) + done + echo " long-polls still in flight after ${LONGPOLL_SETTLE}s: $inflight / $LONGPOLL_N" + ;; +*) fail "unknown PROVOKE '$PROVOKE' (use abend|longpoll)." ;; +esac echo # --- 3. issue P HTTPD ------------------------------------------------------- @@ -201,8 +256,16 @@ clean=$(grep -Ec "HTTPD002I Server is SHUTDOWN|HTTPD060I SHUTDOWN worker" "$wind # run when JES config omits one. if [ "$hasp395" -ne 0 ] || [ "$ief404" -ne 0 ]; then end_seen=1; else end_seen=0; fi +# proof the intended scenario actually ran (mode-specific): +# abend -> a handler faulted (fault marker present in the window) +# longpoll -> at least one long-poll was still parked when P HTTPD landed +case "$PROVOKE" in +abend) provoked=$faultmarks; provoke_desc="fault marker ($FAULT_MARK)" ;; +longpoll) provoked=$inflight; provoke_desc="in-flight long-poll at stop" ;; +esac + echo "-- assertion over captured console window --" -echo " fault markers ($FAULT_MARK) : $faultmarks" +echo " provocation ($PROVOKE) proof : $provoked [$provoke_desc]" echo " S33E : $s33e" echo " __CRTGET ... not found in PPA : $crtget" echo " SVC dump markers : $svcdump" @@ -215,13 +278,17 @@ echo rm -f "$window" # --- 6. verdict ------------------------------------------------------------- -if [ "$faultmarks" -eq 0 ]; then +if [ "$provoked" -eq 0 ]; then cat <