Shutdown crash: S33E on worker DETACH + recovery ESTAE runs C runtime under a torn-down CRT

[mgrossmann]

Summary

On STC shutdown (after a CGI endpoint had been faulting during the session), the address space terminates with an S33E and an SVC dump. Two distinct problems appear in the log; this issue tracks the shutdown lifecycle one (the repeated MVSMF99E is tracked separately — see Follow-up).

Observed

HTTPD060I SHUTDOWN worker(11CC00) TCB(9BB9D8) ...
MVSMF99E Handler abend S0C4 U0000 for PUT /zosmf/restconsoles/consoles/defcn   (x8, same URI)
(~16 s silence)
__CRTGET CRT for TCB(009B29E8) was not found in PPA(000CEFD0)                  (repeated)
ABEND S33E detected for module 00000000  TCB=009B29E8
PSW:00000000 00000000 KEY(0) MODE(SUP) ILC(2) CC(0)
Traceback interrupted, Forward from PPA 002BEFF0
-> address space terminated, SVC dump taken

Root cause

1. Teardown DETACHes tasks without proving they terminated.
cthread_manager_term → dispatch_thread_term → cthread_worker_shutdown → cthread_detach issues DETACH ...,STAE=YES on the escalation path even when the target is still active / already detached / already reclaimed → S33E. The ATTACH supplies ECB=(task->termecb) (posted 0x40000000 when the subtask actually ends), and the join loops wait on it, but the detach decision itself is not gated on it. @@cmterm.c already documents this failure mode:

force-DETACHes a still-active worker (S33E) and the worker's ESTAE then faults on the torn-down save area (nested S0C4).

The ~16 s of silence is the STIMER WAIT escalation loops (@@cmwshu.c waits 50 x 0.10 s per force-detached worker; @@cmterm.c join loops run tens of seconds) trying to drain workers that won't drain.

2. Recovery ESTAEs do runtime-dependent work at termination.
The worker ESTAE (libc370 @@abrpt.c recovery()) is created ESTAE ...,TERM=YES but never inspects SDWACLUP (clibsdwa.h:379 — "ON INDICATES RECOVERY EXIT ONLY TO CLEANUP AND NOT RETRY"), and it calls the C runtime (wtof, sprintf, and try() → __crtget, estae, GETMAIN) for every register (dump_regs) and every save area (dump_sa). Driven during termination under a TCB whose libc370 CRT is already torn down, each try() → __crtget() fails and logs __CRTGET ... not found in PPA — hence the repeated spam. (The zero PSW/regs is correct output for an RTM/DETACH-detected S33E, which carries no faulting-instruction PSW — not a second bug.) mvsMF's per-request ESTAE failed() in @@try.c has the same defect (ignores SDWACLUP, unconditionally requests retry).

This is the same class as mvslovers/mvsmf#101 (SE37 during fclose in ESTAE recovery): a cleanup path that assumes a healthy runtime and takes a secondary fault.

Ordering note

HTTPD100I CONS(0) STOP before the worker shutdowns is expected and correct — that CONS is the operator console / CIB interface, serviced only by the main thread (httpcons.c), and is the trigger for shutdown, not a resource the workers reference during cleanup. The real race is inside worker teardown (detach vs task termination), not the console.

Proposed fix

• Gate DETACH on proof of death. Treat task->termecb & 0x40000000 as the single source of truth for "this TCB is gone"; issue DETACH only after it posts. Make cthread_detach idempotent and owner/termination-checked (verify the TCB is still a subtask of the caller and has posted termecb before detaching; skip otherwise).
• Never STAE=YES-detach a live worker. If a worker won't drain, mark it STUCK, leave its storage, and let address-space termination reclaim it (log the leak) — the same "retain rather than risk use-after-free" tradeoff cthread_manager_term already makes for the dispatch thread.
• Harden the recovery exits. In @@abrpt.c recovery() and @@try.c failed(): check SDWACLUP first; if set, emit at most one WTO from a pre-formatted buffer and return RC=0 ("continue with termination") — no retry, no dump/register walk, no malloc/try/__crtget. Guard the dump path against a missing CRT.

Notes / next steps

• Confirm the detaching TCB (009B29E8) by matching against the HTTPD061I STARTING / HTTPD060I SHUTDOWN TCB(...) log lines (dispatch thread vs main task) — this distinguishes double-detach vs cross-owner vs detach-after-reclaim.
• The recovery-exit hardening likely belongs in libc370 (@@abrpt.c, @@try.c); coordinate with mvsmf#101.
• Follow-up (separate issue): the repeated MVSMF99E on the same URI is a faulting/wedged endpoint whose retries spread across the whole worker pool — the message data is per-client (httpc->env), not a shared field. It may be compounded by a shared grtapp2 handoff (the GRT is inherited/shared down the task tree — @@crtset.c). Instrument grt/&grtapp2/httpc per CGI invocation to decide: different httpc per worker = mvsMF endpoint bug; same grtapp2/httpc across concurrent workers = shared-slot stomp.

[mgrossmann]

Ownership resolved — code fix repointed to libc370

Phase 0 of the v1.0.0 shutdown work (epic mvslovers/mvsmf#177, Workstream A)
confirmed that every file and symbol named in this issue is defined in
mvslovers/libc370 — @@cmterm.c / cthread_manager_term(), @@cmwshu.c /
cthread_worker_shutdown(), dispatch_thread_term(), cthread_detach(),
@@abrpt.c / recovery(), @@try.c / failed(), @@crtset.c /
__crtset(), and __crtget(). httpd vendors none of these @@-files and
overrides no symbol at link time — it only consumes them by autocall from
libc370.a (e.g. the __crtget() call sites in httpd/src/*). So the root
causes here — force-DETACH without proof of termination, and recovery ESTAEs
that run the C runtime under a torn-down CRT — are libc370 code, not httpd
code.

Two distinct libc370 defects own the code fix

The two root causes in the description are separate defects with separate
fixes, tracked by two libc370 tickets:

• DETACH / teardown race → S33E: cthread manager teardown races on shutdown: active worker force-DETACHed (S33E) + nested abend-reporter S0C4 libc370#6 (closed — fixed
  weeks ago). The S33E was nonetheless observed here on 2026-07-22, after that
  fix. Whether that is a stale deployed build, an incomplete fix, or a
  regression is being determined in libc370; this comment does not assert
  which.
• Recovery exits ignore SDWACLUP, drive the C runtime at termination →
  __CRTGET ... not found in PPA spam and secondary faults:
  Recovery exits run the C runtime at termination — check SDWACLUP, emit one WTO, RC=0 libc370#9 (new, open) — canonical for that half.

Epic: mvslovers/mvsmf#177.

Two file-location corrections to the description above

Neither changes the owning repo; both are still libc370:

• dispatch_thread_term() is static in @@cminit.c
  (src/thdmgr/@@cminit.c, around line 470) — not @@cmterm.c.
• cthread_detach() lives in @@ctdet.c (src/clib/@@ctdet.c) — a file
  this issue did not name.

What #122 still tracks (httpd-side)

This issue stays open as the httpd-side tracking and acceptance record:

1. The S33E is observed in httpd's address space — the STC that crashes on
   P HTTPD is HTTPD, so httpd owns confirming the symptom is gone here.
2. The shutdown acceptance test is an httpd-level integration test —
   provoke a handler abend, issue P HTTPD, assert clean termination (no S33E,
   no repeated __CRTGET ... not found in PPA, no SVC dump). It lives in
   httpd's tests/ (tests/shutdown-acceptance.sh) and is the acceptance
   criterion for this issue (epic Workstream D). It is expected to fail on
   the current build and to pass only once the libc370 side is resolved and
   relinked.
3. The grtapp2 / per-CGI follow-up is httpd-specific — the repeated
   MVSMF99E on the same URI, and whether concurrent workers share a
   grtapp2/httpc slot, is httpd/CGI-dispatch behaviour to instrument here,
   independent of the libc370 teardown fix.

Closing waits on: libc370#9 landed and the libc370#6 S33E resolved (stale
build, incomplete fix, or regression — TBD in libc370), both relinked into the
httpd build, and the shutdown acceptance test green against that build.

[mgrossmann]

Reopened. The close was an unintended side effect: PR #123's description contained the phrase "does not close #122", and GitHub's keyword parser matched the "close #122" substring (it does not honour the negation), registering the merged PR as closing this issue. The PR only adds tests/shutdown-acceptance.sh — it does not resolve the underlying bug.

Per the acceptance criteria above, this stays open until libc370#9 lands, the libc370#6 S33E disposition is settled, and tests/shutdown-acceptance.sh is green against the relinked build. The PR description has been corrected to remove the trigger.

[mgrossmann]

Shutdown-crash reproduction — 2026-07-22 findings and corrections

Four attempts today to reproduce the P HTTPD shutdown crash. All four shut
down cleanly. Recording the results and the corrections they force, plus one
question on the acceptance-test assertion.

What was run

#	Provocation	Path exercised	Result
1	/abend serial	worker-level fault → recovery()	clean shutdown
2	mvsMF /zosmf/test?fn=abend serial	CGI-level fault → failed() (12× MVSMF99E confirmed in the log)	clean shutdown
3	fn=abend parallel	faults in flight at P HTTPD	clean shutdown
4	defcn PUT parallel	216 requests, 0 errors, poll loop in flight	clean shutdown

Attempt 2 used the guarded hook from mvslovers/mvsmf#178, built specifically to
drive failed() rather than recovery().

Corrections to the record

1. The ESTAE re-drive theory does not hold. An earlier working theory held
that mvsMF's ESTAE was being re-driven at task termination and printing stale
request data — the source of the repeated MVSMF99E. Attempt 2 tests that
directly. In its fault-injection phase it produced 12× MVSMF99E (one per
CGI abend, failed() firing as intended — confirmed in the log). Those twelve
workers then carried an armed failed() exit into shutdown, and at teardown
none of them emitted a further MVSMF99E. If the re-drive theory were true,
each armed exit would have re-printed at P HTTPD; zero did. The re-drive
explanation is refuted. The follow-up note's attribution stands instead: the
repeated MVSMF99E is a faulting/wedged endpoint whose retries spread across
the worker pool, not a teardown re-drive.

2. The crash is not independently reproducible. It appears to require a
worker that genuinely fails to drain within cthread_worker_shutdown's ~5 s
window, and nothing that can be triggered on demand produces one. The single
known occurrence was under mvslovers/mvsmf#176, when defcn was abending
continuously (6174 requests / 646 errors in that address space) — versus
0 errors across today's runs.

3. libc370#6 is fixed and already in the sysroot; a relink will not clear the
residual. libc370#6 was fixed by libc370 PR #7 (8ce34b3, 2 July) and is
present in the build host's sysroot (libc.a dated 5 July). A relink cannot
address the remaining symptom — PR #7 never touched @@cmwshu.c. The residual
force-DETACH at @@cmwshu.c:64 is being tracked as a new libc370 issue.

4. PR #123's acceptance test cannot be the merge gate for libc370#9. It
exercises recovery() only, and it cannot produce a non-draining worker — the
exact condition #122 needs. It remains valuable as regression protection for
the recovery() path, but a green run must not be read as clearing this
issue. Stating that plainly so it is not misread later.

5. Impact framing (for release scoping). The S33E occurs during shutdown
of an address space that is terminating anyway. It produces an SVC dump and an
ugly console, but no in-flight request or data is lost.

Question — the __CRTGET assertion in tests/shutdown-acceptance.sh

How the assertion is written today: it is zero-tolerance. Line 153 counts
the matches and line 182 fails on any nonzero count:

crtget=$(grep -Ec "__CRTGET CRT for TCB.*was not found in PPA" "$window" ...)
...
if [ "$s33e" -ne 0 ] || [ "$crtget" -ne 0 ] || [ "$svcdump" -ne 0 ]; then   # → FAIL

crtget -ne 0 means none — a single line reports FAIL.

Why that can misfire on a correct fix. libc370#9 is still open; its required
behaviour is "on SDWACLUP clear, guard the dump path against a missing CRT so
it degrades to the single-WTO path." The obvious way to write that guard is
if (__crtget()) — and __crtget() is itself what emits __CRTGET CRT for TCB(...) was not found in PPA(...) when it misses. So if the fix is written
that way:

• SDWACLUP set → early-out, no __crtget() call → zero lines.
• SDWACLUP clear + CRT already gone → the guard call itself emits one line
  before degrading to the single-WTO path.

A correctly fixed build then legitimately produces one __CRTGET line on the
second path, and today's crtget -ne 0 reports FAIL on it.

Answer: yes — the assertion needs relaxing from "none", but note two things
before picking the threshold.

1. It is contingent on how libc370#9 lands. The relaxation is only required if
   the CRT guard emits (as if (__crtget()) would). libc370#9 could instead
   probe CRT presence without emitting — in which case "none" stays achievable
   and no change is needed. That is a live lever on the libc370 side; flagging it
   so the two are decided together, not independently.

2. "At most one" (crtget -gt 1) is not automatically safe either. P HTTPD
   tears down a 3–9 worker pool. If more than one worker runs its
   recovery()/failed() exit on the SDWACLUP-clear + CRT-gone path, each
   emits one line — so a correct fix could show more than one line total, and
   a flat -gt 1 would still FAIL it. The real discriminator is "one line per
   exit" (fixed) vs. "many lines from a single exit's register/save-area walk"
   (the bug), which a flat count cannot perfectly capture.

Relaxing to "at most one" also aligns the script with its own stated intent:
both this issue's acceptance criteria and PR #123's description say "no
repeated __CRTGET", and a single guard-check line is not "repeated."
Zero-tolerance is stricter than the documented contract. But the safe threshold
depends on the final libc370#9 behaviour and on how many exits can hit the
emitting path per shutdown — so the fix and the assertion should be tuned
together rather than blind-swapping -ne 0 for -gt 1.

Not changing the script yet — reporting first per the plan.

[mgrossmann]

Re-scope after the 2026-07-23 relink — recovery-exit half validated, teardown half remains

libc370 was rebuilt with the #9 recovery-exit hardening; httpd and mvsMF were
relinked and deployed. Identical scenario before and after: 4 concurrent
unsol_sync requests (unsol-detect-timeout=20) against
PUT /zosmf/restconsoles/consoles/defcn, then P HTTPD.

The recovery-exit half is fixed — validated on the live system, not just code-reviewed

This is measured against the before/after console, not inferred from the patch:

• Before (11.02): S33E on two wedged workers, a nested S0C4 in
  get_addr (PSW 00000000, module 00000000), and __CRTGET ... not found in PPA spam. Address space ended.
• After (11.35): the worker recovery exits now log
  libc370 recovery: abend during task termination, cleanup only (×4) and
  stop there — no __CRTGET spam, no nested S0C4, no S33E. That is
  exactly the SDWACLUP-checked, single-WTO, RC=0 behaviour libc370#9
  specifies, observed in httpd's own address space.

So libc370#9 can be treated as validated here, on this build, against the
scenario that originally produced the __CRTGET storm.

The teardown half is not fixed — the symptom moved from S33E to SA03

@@cmwshu.c:64 is unchanged, and the address space still ends abnormally.
The 11.35 run ended:

cthread_manager_term(000DE8F0): dispatch thread did not stop,
    storage retained to avoid use-after-free
IEF450I HTTPD - ABEND SA03

S33E is gone; SA03 is what replaced it. Per GC38-1008, SA03 is a task
returning normally while subtasks it attached are still active — the same fact
libc370 reports one line earlier ("dispatch thread did not stop").

The causal chain is now measured, not hypothesised

mvsMF poll loops ignore quiesce (mvsmf#179)
  -> workers keep polling
  -> the dispatch thread waits on them
  -> cthread_manager_term gives up (retains storage to avoid use-after-free)
  -> the main task returns with a live subtask
  -> SA03

Evidence: two D T commands during the hang, at 11.35.07 and 11.35.28
— i.e. 42 s and 63 s after quiesce — both show the poll workers still
active. The workers are not draining; they never see the quiesce.

What to fix first

mvsmf#179 is at the root and should be fixed first. It is small,
mvsMF-local, and needs no libc370 change: make the poll loops honour quiesce so
workers drain, the dispatch thread stops, and the main task returns with no live
subtask. Everything downstream (cthread_manager_term giving up, the SA03)
falls out of that one omission.

Proposed closing criterion (so this does not drift)

This issue should remain open until the shutdown acceptance test
(tests/shutdown-acceptance.sh) reports PASS with a normal address-space
termination — not merely the absence of S33E. The 11.35 run is the reason
the bar has to be stated that way: it had no S33E, no __CRTGET spam, and
no SVC dump, and still ended IEF450I ... ABEND SA03. "No S33E" is now a
necessary-but-insufficient signal.

A companion update to the acceptance test is prepared and under review (not yet
merged): it adds FAIL assertions for IEF450I ... ABEND,
$HASP310 ... TERMINATED AT END OF MEMORY, and SVC-dump markers (IEA911E), and
it confirms the address-space end was actually captured in the window
($HASP395 ... ENDED or IEF404I) before reporting PASS. Those two end records
appear on clean and abnormal ends alike, so they gate log-completeness, not
health — the discriminators are IEF450I / $HASP310 / SVC dump. Once it lands,
a green run will mean the address space terminated normally, not merely that
S33E was absent.

Caveat on the evidence

One run per side so far. The scenario is being repeated 2–3 times; do not read
SA03-vs-S33E as an established pattern yet — it is a single before/after
pair. The direction (recovery spam gone, abnormal end persists via a live
subtask) is consistent, but the count needs to hold up across repeats.

[mgrossmann]

Correction to the earlier __CRTGET assertion recommendation

An earlier comment here recommended relaxing the shutdown test's
__CRTGET ... not found in PPA assertion from "none" to "at most one". That
needs narrowing — and scoped to what this test does, "none" is the correct
bar, so the assertion stays zero-tolerance (unchanged in the pending update).

libc370 PR #10 (the libc370#9 fix) documents two distinct paths through the
guarded recovery exits:

• Termination / shutdown path — SDWACLUP set. RTM drives the
  ESTAE ...,TERM=YES exit with SDWACLUP on; both exits take the early-out
  and call __crtget() zero times, so no __CRTGET line is possible. PR [HTTPD] HTTP Request Parsing härten #10
  states this and recommends the assertion be none on the termination/shutdown
  path.
• Non-termination path — SDWACLUP clear. The guard's lone
  if (__crtget()) fires only for a non-termination abend that hits a missing
  CRT, emitting at most one line (this replaces the old per-probe spam).

This acceptance test provokes /abend then P HTTPD — it drives only the
termination path. Therefore:

• Zero-tolerance (crtget -ne 0 → FAIL) is correct for the paths this test
  exercises. That is a guarantee by design (SDWACLUP set → no __crtget()
  call), not an inference from one run — the 2026-07-23 11.35 run, with zero
  __CRTGET lines, is merely consistent with it.
• The at-most-one case belongs to the SDWACLUP-clear non-termination path,
  which this test never exercises and therefore leaves untested here.

Caution for a later reader: do not carry the zero rule onto a scenario that hits
the SDWACLUP-clear path (a non-termination abend under a torn-down CRT). One
__CRTGET line there is expected behaviour, not a regression — reading it as a
FAIL would be spurious.