From a70aed09948f67b59ba0861c4b461b5acf746b32 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mike=20Gro=C3=9Fmann?= Date: Wed, 22 Jul 2026 16:13:18 +0200 Subject: [PATCH] Add shutdown acceptance test for #122 Adds tests/shutdown-acceptance.sh, the httpd-side acceptance criterion for the shutdown crash tracked in #122: after a request handler faults, P HTTPD must terminate the address space cleanly (no S33E, no repeated "__CRTGET CRT for TCB(...) was not found in PPA(...)", no SVC dump). The test provokes a faulting handler via the built-in /abend path (DC H'0' inside the worker request pipeline), issues P HTTPD, then asserts over the MVS console log. It reports FAIL when the crash signatures appear (the expected result on the current build), PASS on clean shutdown, and INCONCLUSIVE when no handler-fault marker (HTTPD062E or MVSMF99E) is present. Two correctness safeguards: the assertion is windowed to the byte range appended during the run so a pre-existing crash in an append-only console log cannot pin the result at FAIL; and proof-of-fault accepts HTTPD062E (core abend) or MVSMF99E (mvsMF CGI abend). Issuing P HTTPD and capturing the console log are pluggable adapters (manual operator default, cmd for CI), documented in tests/README.md. Refs #122 --- tests/README.md | 103 ++++++++++++++++++ tests/shutdown-acceptance.sh | 202 +++++++++++++++++++++++++++++++++++ 2 files changed, 305 insertions(+) create mode 100644 tests/README.md create mode 100755 tests/shutdown-acceptance.sh diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000..c957a1a --- /dev/null +++ b/tests/README.md @@ -0,0 +1,103 @@ +# httpd integration tests + +Runnable checks against a live HTTPD address space on MVS. Unlike the +`test/*.c` unit tests (built with mbtcheck), these drive the server over the +wire and inspect the MVS console. + +## `shutdown-acceptance.sh` — acceptance test for #122 + +Acceptance criterion for [#122](https://github.com/mvslovers/httpd/issues/122) +(epic [mvsmf#177](https://github.com/mvslovers/mvsmf/issues/177), Workstream D): +after a request handler faults, `P HTTPD` must terminate the address space +cleanly — **no `S33E`**, **no repeated `__CRTGET CRT for TCB(...) was not found +in PPA(...)`**, **no SVC dump**. + +The code fix lives in libc370 (worker DETACH gating + recovery-exit hardening). +This test observes the symptom in httpd's own address space. + +### What it does + +1. Records the current byte offset of `CONSOLE_LOG` (the assertion window + starts here — see "Windowed assertion" below). +2. Provokes a faulting handler: `ABEND_HITS` requests to `ABEND_PATH` + (default `/abend`, whose handler executes `DC H'0'` → S0C1 inside the + worker request pipeline). Repeated hits poison the whole worker pool. +3. Issues `P HTTPD` (see adapters below). +4. Waits up to `WAIT_SECS` for the listener to stop accepting. +5. Asserts over the console-log slice appended during this run. + +### Result + +- **FAIL** — a fault marker is present *and* any of `S33E` / `__CRTGET` spam / + SVC dump appears. This is the #122 bug reproduced and is the **expected + result on the current build**. +- **PASS** — a handler faulted but shutdown was clean. The post-fix target. + A PASS on a build that still carries the libc370 defect is suspicious — the + banner says so. +- **INCONCLUSIVE** — no fault marker (`HTTPD062E` / `MVSMF99E`) in the window, + so no handler actually faulted. The test did not exercise #122; fix the + provocation and re-run. + +Exit codes: `0` pass, `1` fail, `2` config error, `3` inconclusive. + +### Windowed assertion (important) + +`CONSOLE_LOG` is usually an **append-only** Hercules hardcopy log that already +contains `S33E` / `__CRTGET` lines from earlier real crashes. The script +records the log's byte offset *before* provoking and asserts only over bytes +appended afterwards. Without this the test would report FAIL forever — even +after the fix. Do not remove it. + +### Fault marker + +Proof that a handler faulted is `HTTPD062E` **or** `MVSMF99E`: + +- `HTTPD062E` — a **core** abend caught by the worker's `try(serve_client)` + (`src/httpd.c`). This is what `/abend` produces. +- `MVSMF99E` — an **mvsMF CGI** abend caught by mvsMF's inner ESTAE; the + worker `try()` then returns 0, so `HTTPD062E` does not fire. + +Accepting either keeps the test valid whether `ABEND_PATH` is a core path or a +CGI endpoint. + +### Issuing `P HTTPD` and capturing the console log (adapters) + +The repo has no console/SYSLOG tooling, so both are pluggable. + +**`STOP_ADAPTER=manual` (default).** The script pauses; the operator issues +`P HTTPD` at the MVS console and presses Enter. `CONSOLE_LOG` points at a +captured console/hardcopy log (e.g. the Hercules hardcopy log file, or a +SYSLOG extract) that covers the shutdown window. Most portable across +TK4-/TK5/MVSCE/Hercules. + +**`STOP_ADAPTER=cmd`.** Set `STOP_CMD` to a command that issues `P HTTPD` +unattended — for local Hercules, a pipe to the console port; or a one-shot +operator-command REST call. For CI. Note the self-stop race: do **not** issue +`P HTTPD` through an mvsMF console endpoint served *by the httpd under test* — +that address space is terminating, so a follow-up read of live SYSLOG is +unreliable (see libc370#4). Capture `CONSOLE_LOG` out of band. + +### Configuration + +Environment variables (or a sourced `.env`): + +| Variable | Default | Meaning | +|---|---|---| +| `HTTPD_HOST` | `MBT_MVS_HOST` or `127.0.0.1` | host serving the HTTPD listener | +| `HTTPD_PORT` | `8080` | HTTPD listener port | +| `ABEND_PATH` | `/abend` | path whose handler faults | +| `ABEND_HITS` | `12` | number of provocation requests | +| `FAULT_MARK` | `HTTPD062E\|MVSMF99E` | proof-of-fault regex | +| `HTTPD_AUTH` | (unset) | `user:pass` if `LOGIN` gates GET | +| `WAIT_SECS` | `120` | max wait for the AS to end | +| `STOP_ADAPTER` | `manual` | `manual` or `cmd` | +| `STOP_CMD` | (unset) | command to issue `P HTTPD` when `cmd` | +| `CONSOLE_LOG` | (required) | captured console/hardcopy log to assert over | + +### Example + +```sh +CONSOLE_LOG=/var/hercules/mvs/hardcopy.log \ +HTTPD_HOST=mvs HTTPD_PORT=8080 \ +sh tests/shutdown-acceptance.sh +``` diff --git a/tests/shutdown-acceptance.sh b/tests/shutdown-acceptance.sh new file mode 100755 index 0000000..ff8f037 --- /dev/null +++ b/tests/shutdown-acceptance.sh @@ -0,0 +1,202 @@ +#!/bin/sh +# shutdown-acceptance.sh — acceptance test for mvslovers/httpd#122 +# +# Verifies that stopping HTTPD (P HTTPD) AFTER a request handler has faulted +# terminates the address space cleanly — no S33E, no repeated "__CRTGET CRT +# for TCB(...) was not found in PPA(...)", no SVC dump. +# +# Background: on shutdown, worker teardown DETACHes tasks without proving they +# terminated (S33E), and the worker recovery ESTAE runs the C runtime under a +# torn-down CRT (__CRTGET spam). The code fix lives in libc370. This test is +# the httpd-side acceptance criterion for #122 — it observes the failure in +# httpd's own address space. +# +# EXPECTED RESULT ON THE CURRENT (PRE-FIX) BUILD: FAIL. +# A PASS today does not mean "green" — it means the provocation did not leave a +# worker in the post-fault state #122 needs. See the result banner at the end. +# +# --------------------------------------------------------------------------- +# TWO THINGS THAT SILENTLY BREAK THIS TEST — read before editing: +# +# 1. WINDOWED ASSERTION. CONSOLE_LOG is typically an append-only Hercules +# hardcopy log that ALREADY contains S33E / __CRTGET lines from earlier +# real crashes. Grepping the whole file would report FAIL forever — even +# after the fix — and look exactly like the fix not working. This script +# records the byte offset of CONSOLE_LOG immediately before provoking and +# asserts ONLY over bytes appended after that point. Do not remove this. +# +# 2. FAULT MARKER. Proof-of-fault is HTTPD062E *or* MVSMF99E. httpd emits +# HTTPD062E for a CORE abend caught by the worker's try(serve_client) +# (httpd.c:682) — this is what /abend produces. An mvsMF CGI abend is +# caught by mvsMF's inner ESTAE instead and logs MVSMF99E, so the worker +# try() returns 0 and HTTPD062E never fires. Accepting either keeps the +# test valid whether ABEND_PATH is core (/abend) or a CGI endpoint. +# --------------------------------------------------------------------------- +# Configuration (override via environment or a sourced .env) +# --------------------------------------------------------------------------- +# HTTPD_HOST host serving the HTTPD listener (default: MBT_MVS_HOST or 127.0.0.1) +# HTTPD_PORT HTTPD listener port to attack + poll (default: 8080) +# ABEND_PATH path whose handler faults (default: /abend) +# ABEND_HITS how many times to hit it (poison pool) (default: 12) +# FAULT_MARK regex proving a handler faulted (default: HTTPD062E|MVSMF99E) +# HTTPD_AUTH optional "user:pass" if LOGIN gates GET (default: unset) +# WAIT_SECS max seconds to wait for the AS to end (default: 120) +# +# STOP_ADAPTER how "P HTTPD" is issued: manual | cmd (default: manual) +# STOP_CMD command line to issue P HTTPD when STOP_ADAPTER=cmd +# (e.g. a Hercules console pipe, or a one-shot operator-command +# REST call — see tests/README.md). Must return promptly. +# +# CONSOLE_LOG REQUIRED. Path to the MVS console / hardcopy log (SYSLOG +# extract, or the Hercules hardcopy log). The script asserts +# over the slice appended during THIS run. It must cover the +# window from "P HTTPD" through address-space termination. +# --------------------------------------------------------------------------- + +set -u + +# --- load .env if present (for HTTPD_HOST default only) --------------------- +if [ -f "${ENV_FILE:-.env}" ]; then + # shellcheck disable=SC1090 + . "${ENV_FILE:-.env}" 2>/dev/null || true +fi + +HTTPD_HOST=${HTTPD_HOST:-${MBT_MVS_HOST:-127.0.0.1}} +HTTPD_PORT=${HTTPD_PORT:-8080} +ABEND_PATH=${ABEND_PATH:-/abend} +ABEND_HITS=${ABEND_HITS:-12} +FAULT_MARK=${FAULT_MARK:-HTTPD062E|MVSMF99E} +WAIT_SECS=${WAIT_SECS:-120} +STOP_ADAPTER=${STOP_ADAPTER:-manual} + +fail() { printf '%s\n' "$*" >&2; exit 2; } + +[ -n "${CONSOLE_LOG:-}" ] || fail "CONSOLE_LOG is required (path to the captured console/hardcopy log)." +[ -r "${CONSOLE_LOG}" ] || fail "CONSOLE_LOG '$CONSOLE_LOG' is not readable." + +CURL="curl -s -S -o /dev/null -w %{http_code} --max-time 10" +[ -n "${HTTPD_AUTH:-}" ] && CURL="$CURL -u $HTTPD_AUTH" +BASE="http://${HTTPD_HOST}:${HTTPD_PORT}" + +echo "== httpd#122 shutdown acceptance ==" +echo " target : $BASE" +echo " abend : $ABEND_PATH x $ABEND_HITS" +echo " stop : $STOP_ADAPTER" +echo " log : $CONSOLE_LOG" +echo + +# --- 1. mark the log window START (byte offset) — see note #1 above --------- +# Everything already in CONSOLE_LOG (including prior real crashes) is BEFORE +# this offset and is excluded from the assertion. +log_offset=$(wc -c < "$CONSOLE_LOG") +log_offset=$(printf '%s' "$log_offset" | tr -d ' ') +echo "-- console-log window starts at byte offset $log_offset --" +echo + +# --- 2. provoke: fault the handler enough times to poison the worker pool --- +echo "-- provoking handler abend ($ABEND_PATH) --" +faulted=0 +i=0 +while [ "$i" -lt "$ABEND_HITS" ]; do + i=$((i + 1)) + code=$($CURL "${BASE}${ABEND_PATH}" 2>/dev/null) || code="conn-reset" + # A faulting worker drops the connection or returns 5xx; a clean 200/404 + # means the handler did NOT fault (wrong path / already handled otherwise). + case "$code" in + 5*|conn-reset|000) faulted=$((faulted + 1)) ;; + esac + printf ' hit %2d -> %s\n' "$i" "$code" +done +echo " handler-fault responses: $faulted / $ABEND_HITS" +echo + +# --- 3. issue P HTTPD ------------------------------------------------------- +echo "-- issuing P HTTPD --" +case "$STOP_ADAPTER" in + cmd) + [ -n "${STOP_CMD:-}" ] || fail "STOP_ADAPTER=cmd requires STOP_CMD." + echo " \$ $STOP_CMD" + sh -c "$STOP_CMD" || fail "STOP_CMD failed." + ;; + manual) + echo " >>> Issue P HTTPD at the MVS operator console now." + echo " >>> Ensure it is written to CONSOLE_LOG ($CONSOLE_LOG)." + printf " >>> Press Enter once the address space has ended... " + read ack_dummy + ;; + *) fail "unknown STOP_ADAPTER '$STOP_ADAPTER' (use manual|cmd)." ;; +esac +echo + +# --- 4. wait for the address space to end (listener stops accepting) -------- +# #122 shows ~16s of STIMER silence before termination; give it room. +echo "-- waiting up to ${WAIT_SECS}s for the listener to stop --" +ended=0 +w=0 +while [ "$w" -lt "$WAIT_SECS" ]; do + if ! $CURL "${BASE}/" >/dev/null 2>&1; then + ended=1; break + fi + w=$((w + 3)); sleep 3 +done +[ "$ended" -eq 1 ] && echo " listener closed after ~${w}s" || echo " listener still up after ${WAIT_SECS}s" +echo + +# --- 5. assert over the console-log slice captured for THIS run — note #1 --- +window=$(mktemp 2>/dev/null || echo "/tmp/httpd122.$$") +tail -c +"$((log_offset + 1))" "$CONSOLE_LOG" > "$window" 2>/dev/null || cp "$CONSOLE_LOG" "$window" + +# proof the provocation actually faulted a worker — HTTPD062E (core) or MVSMF99E (CGI) +faultmarks=$(grep -Ec "$FAULT_MARK" "$window" 2>/dev/null || true) +# failure signatures from #122 +s33e=$(grep -c "S33E" "$window" 2>/dev/null || true) +crtget=$(grep -Ec "__CRTGET CRT for TCB.*was not found in PPA" "$window" 2>/dev/null || true) +svcdump=$(grep -Eic "SVC dump|SDUMP|IEA911E|IEA794I" "$window" 2>/dev/null || true) +# clean-shutdown positive markers (httpd.c:335 / :701) +clean=$(grep -Ec "HTTPD002I Server is SHUTDOWN|HTTPD060I SHUTDOWN worker" "$window" 2>/dev/null || true) + +: "${faultmarks:=0}" "${s33e:=0}" "${crtget:=0}" "${svcdump:=0}" "${clean:=0}" + +echo "-- assertion over captured console window --" +echo " fault markers ($FAULT_MARK) : $faultmarks" +echo " S33E : $s33e" +echo " __CRTGET ... not found in PPA : $crtget" +echo " SVC dump markers : $svcdump" +echo " clean-shutdown markers : $clean" +echo + +rm -f "$window" + +# --- 6. verdict ------------------------------------------------------------- +if [ "$faultmarks" -eq 0 ]; then + cat <