Skip to content

Add longpoll provocation mode to the shutdown acceptance test #126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mgrossmann merged 2 commits into from
Jul 23, 2026
Merged

Add longpoll provocation mode to the shutdown acceptance test #126

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dd097d8
tests: add longpoll provocation mode to exercise the parked-poll shut…
mgrossmann Jul 23, 2026
24365d7
tests: fix longpoll example URL and note manual-mode timing
mgrossmann Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 51 additions & 8 deletions tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,51 @@ This test observes the symptom in httpd's own address space.

1. Records the current byte offset of `CONSOLE_LOG` (the assertion window
starts here — see "Windowed assertion" below).
2. Provokes a faulting handler: `ABEND_HITS` requests to `ABEND_PATH`
(default `/abend`, whose handler executes `DC H'0'` → S0C1 inside the
worker request pipeline). Repeated hits poison the whole worker pool.
2. Provokes the pre-stop state under test (`PROVOKE`, see "Provocation modes"):
either faulting handlers (`abend`) or workers parked in an in-flight long
poll (`longpoll`, the `#179` / `SA03` path).
3. Issues `P HTTPD` (see adapters below).
4. Waits up to `WAIT_SECS` for the listener to stop accepting.
5. Asserts over the console-log slice appended during this run.

### Provocation modes (`PROVOKE`)

Which shutdown path the run exercises:

- **`abend`** (default) — `ABEND_HITS` requests to `ABEND_PATH` (default
`/abend`, whose handler executes `DC H'0'` → S0C1). Handlers fault *fast*, so
this drives libc370's **recovery-drain** path. Proof-of-scenario is a fault
marker (`HTTPD062E` / `MVSMF99E`) in the window.
- **`longpoll`** — launches `LONGPOLL_N` (default 4) concurrent, blocking
long-poll requests (`LONGPOLL_CMD`) and checks that at least one is *still in
flight* `LONGPOLL_SETTLE` s later, then issues `P HTTPD` while they are parked.
This is the only mode that reproduces `#179` / the `SA03`: a worker parked in a
long poll cannot drain inside libc370's ~5 s window. Proof-of-scenario is the
in-flight count — **not** a fault marker (a clean parked poll faults nothing).

> A green **`abend`** run does **not** establish the `#179` fix — it tests a
> different path. Use **`longpoll`** to gate `#179` (mvslovers/mvsmf#179).

Example `LONGPOLL_CMD` — one synchronous unsolicited-message detection PUT that
blocks up to 60 s (mvsMF console **issue-command** endpoint,
`PUT /zosmf/restconsoles/consoles/{console-name}` — *not* the `/solmsgs` collect
sub-resource):

```sh
LONGPOLL_CMD='curl -s -o /dev/null -u USER:PASS -X PUT \
-H "Content-Type: application/json" \
-d "{\"cmd\":\"D T\",\"unsol-key\":\"IEE136I\",\"unsol-detect-sync\":\"Y\",\"unsol-detect-timeout\":\"60\"}" \
http://HOST:PORT/zosmf/restconsoles/consoles/defcn'
```

The long poll must **still be running when `P HTTPD` is issued** — that is the
whole point. The in-flight count is measured `LONGPOLL_SETTLE` s after launch,
but the stop lands later. In `STOP_ADAPTER=cmd` mode the stop follows
immediately, so a `unsol-detect-timeout` of ~60 s is ample. In `manual` mode the
operator's reaction time is unbounded: set `unsol-detect-timeout` comfortably
above how long you expect to take typing `P HTTPD`, or the polls will have
already returned (no worker parked → INCONCLUSIVE, not a real reproduction).

### Result

- **FAIL** — a fault marker is present *and* either a crash signature (`S33E` /
Expand All @@ -43,11 +81,12 @@ This test observes the symptom in httpd's own address space.
that still carries the libc370 defect is suspicious — the banner says so.
`HTTPD002I` / `HTTPD060I` worker-shutdown WTOs corroborate a clean drain but do
**not** gate the result; a PASS with none of them present emits a CAUTION.
- **INCONCLUSIVE** — either no fault marker (`HTTPD062E` / `MVSMF99E`) in the
window (no handler faulted), or no crash/abnormal signature yet the
address-space end was not captured (`$HASP395 ... ENDED` / `IEF404I` both
absent — a truncated log looks the same as a clean end). Fix the provocation
or extend the capture and re-run.
- **INCONCLUSIVE** — either the intended scenario didn't run (in `abend`, no
fault marker `HTTPD062E` / `MVSMF99E`; in `longpoll`, no long-poll still in
flight at `P HTTPD`), or no crash/abnormal signature yet the address-space end
was not captured (`$HASP395 ... ENDED` / `IEF404I` both absent — a truncated
log looks the same as a clean end). Fix the provocation or extend the capture
and re-run.

Exit codes: `0` pass, `1` fail, `2` config error, `3` inconclusive.

Expand Down Expand Up @@ -102,6 +141,10 @@ Environment variables (or a sourced `.env`):
| `HTTPD_AUTH` | (unset) | `user:pass` if `LOGIN` gates GET |
| `WAIT_SECS` | `120` | max wait for the AS to end |
| `JOBNAME` | `HTTPD` | MVS jobname of the HTTPD STC (scopes the end-capture assertion) |
| `PROVOKE` | `abend` | pre-stop scenario: `abend` (fault handlers) or `longpoll` (park workers) |
| `LONGPOLL_CMD` | (required for `longpoll`) | one blocking long-poll request, run `LONGPOLL_N`× concurrently |
| `LONGPOLL_N` | `4` | `longpoll`: concurrent in-flight long-polls |
| `LONGPOLL_SETTLE` | `3` | `longpoll`: seconds to let them park before `P HTTPD` |
| `STOP_ADAPTER` | `manual` | `manual` or `cmd` |
| `STOP_CMD` | (unset) | command to issue `P HTTPD` when `cmd` |
| `CONSOLE_LOG` | (required) | captured console/hardcopy log to assert over |
Expand Down
121 changes: 94 additions & 27 deletions tests/shutdown-acceptance.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,16 @@
# the authoritative signal is the address-space end (note 3). The SA03 run
# produced zero HTTPD060I, so a PASS with zero of them emits a CAUTION, not
# a FAIL. Do not turn these into a hard requirement.
#
# 5. PROVOCATION MODE decides WHICH shutdown path you test. Default
# PROVOKE=abend faults handlers -- that is the recovery-drain path
# (libc370#9), NOT a worker parked in a long poll. The #179 / SA03 crash
# needs a worker still IN FLIGHT in a long poll when P HTTPD lands, which a
# fast fault cannot create; PROVOKE=longpoll launches $LONGPOLL_N concurrent
# blocking long-polls (LONGPOLL_CMD, e.g. an unsol-detect-sync console PUT)
# and proves >=1 is still parked at stop. Proof-of-scenario is that in-flight
# count, NOT a fault marker -- a clean parked poll faults nothing. A green
# abend run does NOT establish the #179 fix; use longpoll for that.
# ---------------------------------------------------------------------------
# Configuration (override via environment or a sourced .env)
# ---------------------------------------------------------------------------
Expand All @@ -66,6 +76,16 @@
# WAIT_SECS max seconds to wait for the AS to end (default: 120)
# JOBNAME MVS jobname of the HTTPD started task (default: HTTPD)
#
# PROVOKE pre-stop scenario: abend | longpoll (default: abend)
# abend -> fault handlers (recovery-drain path; libc370#9).
# longpoll -> park workers in an in-flight long poll at P HTTPD
# (the #179 / SA03 path a fast fault cannot create).
# LONGPOLL_CMD PROVOKE=longpoll, REQUIRED: one blocking long-poll request
# (e.g. an unsol-detect-sync console PUT). Run $LONGPOLL_N times,
# concurrently, in the background — see tests/README.md.
# LONGPOLL_N PROVOKE=longpoll: concurrent long-polls (default: 4)
# LONGPOLL_SETTLE PROVOKE=longpoll: seconds to let them park pre-stop (default: 3)
#
# STOP_ADAPTER how "P HTTPD" is issued: manual | cmd (default: manual)
# STOP_CMD command line to issue P HTTPD when STOP_ADAPTER=cmd
# (e.g. a Hercules console pipe, or a one-shot operator-command
Expand Down Expand Up @@ -93,6 +113,9 @@ FAULT_MARK=${FAULT_MARK:-HTTPD062E|MVSMF99E}
WAIT_SECS=${WAIT_SECS:-120}
STOP_ADAPTER=${STOP_ADAPTER:-manual}
JOBNAME=${JOBNAME:-HTTPD}
PROVOKE=${PROVOKE:-abend}
LONGPOLL_N=${LONGPOLL_N:-4}
LONGPOLL_SETTLE=${LONGPOLL_SETTLE:-3}

fail() { printf '%s\n' "$*" >&2; exit 2; }

Expand All @@ -104,10 +127,15 @@ CURL="curl -s -S -o /dev/null -w %{http_code} --max-time 10"
BASE="http://${HTTPD_HOST}:${HTTPD_PORT}"

echo "== httpd#122 shutdown acceptance =="
echo " target : $BASE"
echo " abend : $ABEND_PATH x $ABEND_HITS"
echo " stop : $STOP_ADAPTER"
echo " log : $CONSOLE_LOG"
echo " target : $BASE"
echo " provoke : $PROVOKE"
if [ "$PROVOKE" = abend ]; then
echo " abend : $ABEND_PATH x $ABEND_HITS"
else
echo " longpoll: $LONGPOLL_N concurrent, settle ${LONGPOLL_SETTLE}s"
fi
echo " stop : $STOP_ADAPTER"
echo " log : $CONSOLE_LOG"
echo

# --- 1. mark the log window START (byte offset) — see note #1 above ---------
Expand All @@ -118,21 +146,48 @@ log_offset=$(printf '%s' "$log_offset" | tr -d ' ')
echo "-- console-log window starts at byte offset $log_offset --"
echo

# --- 2. provoke: fault the handler enough times to poison the worker pool ---
echo "-- provoking handler abend ($ABEND_PATH) --"
faulted=0
i=0
while [ "$i" -lt "$ABEND_HITS" ]; do
i=$((i + 1))
code=$($CURL "${BASE}${ABEND_PATH}" 2>/dev/null) || code="conn-reset"
# A faulting worker drops the connection or returns 5xx; a clean 200/404
# means the handler did NOT fault (wrong path / already handled otherwise).
case "$code" in
5*|conn-reset|000) faulted=$((faulted + 1)) ;;
esac
printf ' hit %2d -> %s\n' "$i" "$code"
done
echo " handler-fault responses: $faulted / $ABEND_HITS"
# --- 2. provoke: put the worker pool into the pre-stop state under test -----
# abend -> handlers fault (fast; the recovery-drain path, libc370#9).
# longpoll -> workers PARKED in an in-flight long poll still running when P HTTPD
# lands — the #179 / SA03 path a fast fault cannot reproduce.
LONGPOLL_PIDS=""
inflight=0
case "$PROVOKE" in
abend)
echo "-- provoking handler abend ($ABEND_PATH x $ABEND_HITS) --"
faulted=0
i=0
while [ "$i" -lt "$ABEND_HITS" ]; do
i=$((i + 1))
code=$($CURL "${BASE}${ABEND_PATH}" 2>/dev/null) || code="conn-reset"
# A faulting worker drops the connection or returns 5xx; a clean 200/404
# means the handler did NOT fault (wrong path / already handled otherwise).
case "$code" in
5*|conn-reset|000) faulted=$((faulted + 1)) ;;
esac
printf ' hit %2d -> %s\n' "$i" "$code"
done
echo " handler-fault responses: $faulted / $ABEND_HITS"
;;
longpoll)
[ -n "${LONGPOLL_CMD:-}" ] || fail "PROVOKE=longpoll requires LONGPOLL_CMD (one blocking long-poll request)."
echo "-- launching $LONGPOLL_N concurrent long-polls (must be in flight at P HTTPD) --"
trap '[ -n "$LONGPOLL_PIDS" ] && kill $LONGPOLL_PIDS 2>/dev/null' EXIT INT TERM
i=0
while [ "$i" -lt "$LONGPOLL_N" ]; do
i=$((i + 1))
sh -c "$LONGPOLL_CMD" >/dev/null 2>&1 &
LONGPOLL_PIDS="$LONGPOLL_PIDS $!"
done
# let the requests reach and PARK in the poll loop before we stop the server
sleep "$LONGPOLL_SETTLE"
for p in $LONGPOLL_PIDS; do
kill -0 "$p" 2>/dev/null && inflight=$((inflight + 1))
done
echo " long-polls still in flight after ${LONGPOLL_SETTLE}s: $inflight / $LONGPOLL_N"
;;
*) fail "unknown PROVOKE '$PROVOKE' (use abend|longpoll)." ;;
esac
echo

# --- 3. issue P HTTPD -------------------------------------------------------
Expand Down Expand Up @@ -201,8 +256,16 @@ clean=$(grep -Ec "HTTPD002I Server is SHUTDOWN|HTTPD060I SHUTDOWN worker" "$wind
# run when JES config omits one.
if [ "$hasp395" -ne 0 ] || [ "$ief404" -ne 0 ]; then end_seen=1; else end_seen=0; fi

# proof the intended scenario actually ran (mode-specific):
# abend -> a handler faulted (fault marker present in the window)
# longpoll -> at least one long-poll was still parked when P HTTPD landed
case "$PROVOKE" in
abend) provoked=$faultmarks; provoke_desc="fault marker ($FAULT_MARK)" ;;
longpoll) provoked=$inflight; provoke_desc="in-flight long-poll at stop" ;;
esac

echo "-- assertion over captured console window --"
echo " fault markers ($FAULT_MARK) : $faultmarks"
echo " provocation ($PROVOKE) proof : $provoked [$provoke_desc]"
echo " S33E : $s33e"
echo " __CRTGET ... not found in PPA : $crtget"
echo " SVC dump markers : $svcdump"
Expand All @@ -215,13 +278,17 @@ echo
rm -f "$window"

# --- 6. verdict -------------------------------------------------------------
if [ "$faultmarks" -eq 0 ]; then
if [ "$provoked" -eq 0 ]; then
cat <<EOF
RESULT: INCONCLUSIVE
The console window shows no handler-fault marker ($FAULT_MARK), so no handler
faulted. This test did NOT exercise #122. Check ABEND_PATH, that GET is not
gated by LOGIN (set HTTPD_AUTH), and that CONSOLE_LOG really captured this run,
then re-run.
No proof the intended scenario ran (${provoke_desc} absent), so this run did NOT
exercise the shutdown path under test.
abend : no handler faulted — check ABEND_PATH, LOGIN gating (HTTPD_AUTH),
and that CONSOLE_LOG captured this run.
longpoll : no long-poll was still in flight at P HTTPD — the requests finished
too fast. Raise the endpoint timeout, LONGPOLL_N, or LONGPOLL_SETTLE
so a worker is genuinely parked at stop.
Then re-run.
EOF
exit 3
fi
Expand All @@ -230,7 +297,7 @@ if [ "$s33e" -ne 0 ] || [ "$crtget" -ne 0 ] || [ "$svcdump" -ne 0 ] \
|| [ "$abend" -ne 0 ] || [ "$hasp310" -ne 0 ]; then
cat <<EOF
RESULT: FAIL
The address space did not terminate cleanly after a faulted handler + P HTTPD.
The address space did not terminate cleanly after the provocation + P HTTPD.
A crash signature (S33E / __CRTGET spam / SVC dump) and/or an ABNORMAL address-
space end (IEF450I ... ABEND, or "\$HASP310 ... TERMINATED AT END OF MEMORY")
appeared in the window — see the counts above.
Expand All @@ -257,7 +324,7 @@ fi

cat <<EOF
RESULT: PASS
A handler faulted ($FAULT_MARK present) yet shutdown was clean: no S33E, no
The provocation ran ($provoke_desc) yet shutdown was clean: no S33E, no
__CRTGET spam, no SVC dump, and no abnormal address-space end (no IEF450I ABEND,
no \$HASP310 EOM). The address-space end was captured in the window, so this is a
real clean termination and not a truncated log.
Expand Down