Notice the stop — the handler was the right shape and did nothing

[makseq]

The claim, and what was actually wrong

This started from "node.py does not handle a stop". It does, and has since the
rewrite — a handler on SIGTERM and SIGINT that sets a flag and nothing more, the flag
checked between units of work, the inventory at module scope, a marker written last with
status: "cancelled" and exit_code: 20 agreeing with the code the process returns.
Every line of the shape docs/AUTHORING.md prescribes was there. The
gap was somewhere else, and there were two of them.

1. The mechanism that made the flag readable did nothing

A flag cannot be read by a process parked in a socket call, and this file knew that: it
kept a ledger of transfers in flight and its handler closed them. Neither half worked,
and both were measured rather than argued:

What the file did	What was measured
put the response on the ledger	a response exists only once the store has begun to answer, so during the wait that matters the ledger is empty
called close() on it	mid-read that raises RuntimeError: reentrant call inside <_io.BufferedReader> inside the handler, where contextlib.suppress swallows it, and the read waits out its whole timeout regardless

So the real stop latency was the socket timeout: 12.2 s stopped during a held-open
download, 10.2 s during an upload, and 25.2 s against a store that never answers —
out of a nominal thirty that PROTOCOL.md is explicit
nobody is promised.

The repair registers the connection when its socket is created — before a byte of a
request is sent — and the handler calls shutdown(SHUT_RDWR), which makes the pending call
return at once. Both schemes are covered (plain HTTP is what a local demo and this suite
exercise; TLS is what every presigned URL in production uses, and a fix covering only the
first would be invisible where it matters). Measured after: 0.22 s and 0.24 s. One
thing is exempt on purpose: once the marker is being written nothing may abandon it.

That is one further act inside a signal handler, where AUTHORING says to set a flag and do
no work "and especially network work". The ambiguity is resolved in a comment on the
handler rather than ignored — a socket shutdown starts nothing, waits for nothing and
cannot block; the flag is set first and the shutdown's failure is ignored, so a handler
that fails to abandon anything leaves the step exactly as stopped as before.

2. The harness never asked for the receipt

Four cancellation tests, and not one required a marker to exist. Measured, not
supposed: a copy of node.py that writes no marker on the stop path still passes three of
the four; the fourth catches it only because that scenario has an object already landed.
Stopped before it produces anything, this suite had nothing to say — and the marker is the
only inventory salvage can publish from.

Four new tests, all conforms_today, all basis_reference_quality:

• test_a_stopped_step_leaves_a_receipt_and_says_it_was_stopped
• test_the_receipt_of_a_stopped_step_carries_the_code_the_process_returned
• test_a_stopped_step_claims_no_object_the_store_never_received
• test_a_stop_is_noticed_without_waiting_for_the_transfer_it_landed_in

None may carry basis_contract. external/contract.py does say "A cancelled run still
writes a marker", but it says it while explaining what a field means, and everything the
platform does with an absent marker here treats it as ordinary:
_salvage_what_the_step_produced publishes "what a FAILED or cancelled step managed to
write" and returns nothing when there is none, and collection reports "No completion marker
was written…" as a finding rather than a refusal. A marker is required only after a
reported success. Exit 20 is the same shape — _classify answers cancelled either way;
read the other way round, 20 is the only signal that says "stopped" rather than "broken"
when the platform was not the party that asked.

Review round: the same defect, twice more, in the same file

Review found that the fix above reproduced its own defect one layer up. "Not on the
ledger during the wait that matters" had been fixed for the wait that had been measured,
and there are four waits on this path, reached through different objects:

The wait	Was	Is
getting a connection — DNS, TCP, TLS handshake	governed by the 25 s transfer timeout, and uninterruptible: ssl detaches the plain socket while wrapping it, so shutting it down raises Bad file descriptor and the handshake runs to the timeout anyway (measured)	bounded by its own 3 s budget, with the flag re-checked the moment the call returns. 25.2 s → 3.2 s
waiting for the store to begin answering	fixed in the first round	unchanged
reading the body	ledger empty — http.client closes the connection as soon as it parses the headers of a Connection: close response (urllib sets it on every request) and the close() override took the entry off	no close() override; one request in flight at a time, so the next connection replaces the entry. 24.7 s → 0.3 s
waiting for an upload to be acknowledged	fixed in the first round	unchanged

The TLS half also closed a hole in what the suite can see at all: production is https-only,
this harness's store is plain http, so the mechanism had been proved against the wrong
protocol. It needs no certificate authority and no openssl — a handshake stalls
before any certificate is offered, so a listener that accepts and says nothing is enough.
conformance/stalling.py is that listener, and it also produces the mid-body case, which
the store's hooks cannot (they fire while a request is still being authorised, before a
byte of the response exists).

And the exception meant for a second stop was swallowing the first

"Nothing may abandon the receipt" was applied to every marker, including one claiming
success. A stop landing while that was in flight changed nothing: flag set, upload left
alone, upload lands, step returns 0 — succeeded inside a launch the orchestrator
records as cancelled. Deterministic, and measured: exit 0, receipts ['succeeded'].

The protection is now asymmetric, which is the argument that was made for it originally. A
receipt reporting a failure or a stop cannot be made worse by being cut short and is the
run's only account: protected. A receipt claiming success is the one document a stop can
turn into a lie: not protected — the handler cuts it and the run writes the cancellation it
has become, carrying the same inventory, so salvage publishes exactly as much as a success
would have. The flag is checked once more after that write, before 0 is returned.

Three more tests, same labels: a stop mid-body, a stop during a TLS handshake (bounded, not
cut — and it says so, with its own threshold), and a stop during the success receipt.

Re-review round: a retraction, and the same lesson a third time

The asymmetric protection I argued for above is withdrawn — the reviewer was right.
Leaving a success receipt abandonable does not prevent the lie; it makes which document
survives unknowable. Once the store has the whole body it may commit it and cutting the
socket revokes nothing, so the cancellation written next is a second write to the same
key that can overlap the first, and nothing defines which of two overlapping writes wins.
This harness shows it: its delay hook holds the first commit back, the cancellation lands
first, and the receipt the store serves says succeeded — with the process exiting 20
beside it, which is worse than the defect. The test written for that design looked away
from the value the store serves and blamed the hook, which is the tell.

Now: the receipt is protected whatever it says, the check happens after it, and the
correction begins only once the first write has finished. Sequential writes have an order.
The defence became observable in the process — the mutation that removes the check was
green last round because abandonment covered for it, and is red now.

And the "bounded" wait was not bounded. Three seconds was not an elapsed deadline and
could not be: socket.create_connection resolves the name before there is a socket to
time, then applies the number separately to each address. Measured in a container:

Stimulus	Behaviour
one silently-dropped address, timeout=4	4.0 s
the same name on three such addresses, timeout=4	12.0 s — the number spent three times
a lookup against a resolver that receives every query and answers none	40.6 s, the step's budget applying to none of it

One elapsed deadline now covers the lookup, every address and the handshake; the lookup is
bounded by the only thread in the file, because getaddrinfo takes no timeout at all.
After: 10.4 s and 10.5 s. The value moved 3 s → 10 s, away from what made tests
quick: a deadline firing on a healthy-but-slow connect kills the whole job, because nothing
retries an external step automatically.

Three stimuli were built and thrown away before these tests measured anything — TEST-NET-3
fails in 0.1 s, an unassigned address on the container's own subnet fails at the kernel's
~3 s ARP timeout, an unroutable resolver fails in 0.4 s. Each made its test pass against a
node with no bound at all. What works is a peer that receives and stays silent. The
multi-address test now verifies its stimulus really hangs on this machine and skips if it
does not.

The tests also stopped synchronising on time. The TLS test waited on accept(), which
is released before the client has sent its hello — so an unbounded handshake could pass on
a lucky schedule — and the mid-body test slept half a second. The listener now reads before
it sends, and sends more than any buffer holds, so it announces only once a client has
spoken and, where there is a body, only once the client has consumed it. A harness
self-test holds the instrument to exactly that.

Round four: the correction could not be sequenced either — and the platform settled it

The remedy from the last round fails on a fact node.py already documents about its own
uploads: a transport failure is ambiguous, and a store may accept a body after the
client that sent it has gone. Reproduced by holding the receipt's upload for twelve seconds
against a step that gives up at ten — the cancellation lands first, the timed-out success
commits last, and the run ends succeeded beside exit 20. Sequential calls are not
sequential commits.

So the design got smaller, not a third mechanism. One receipt, or none: the flag is
read once before the document is composed, the write is protected and bounded, and if it
fails nothing else is written to that name. The exit code is decided with the document and
never revised.

What makes that safe was measured on the platform, not assumed

Question	Answer (at origin/master)
What decides the outcome?	The orchestrator's own journal — _TERMINAL_LAUNCH_OUTCOMES keyed on launch state (pipelines/external_finalize.py:254-260, :627-636)
Can a marker claim success?	No — it can only veto one (:1371-1375)
Are a stopped run's objects delivered?	No — salvaged as diagnostics, role='logs', no payload_kind (:3213-3216, :3242-3243); a cancellation racing a collection demotes even verified output (:2769-2770)
Does anything cascade?	No — behind the CAS a cancellation wins (:2645-2651); _cancel_execution never calls it
What does an operator read?	Our sentence, then the marker's exit_code and error — never its status (:2384-2394)
Does exit 0 override?	No — fence, hard stop, deadline and cancellation are tested first (agent/runner.py:2829-2843)

So the residual state is not one the platform can act on. A step that finished its work
and was interrupted while reporting it has genuinely succeeded; the stop arrived late.

The receipt needed a clock, and the deadline leaked through a proxy

Nothing may abandon the receipt, so nothing but elapsed time can end it — and
UPLOAD_TIMEOUT_S bounds silence. Against a store that dribbles the answer out over
forty seconds, never idle: 39.7 s without a deadline, past the whole grace. With one:
20 s, sized against what the platform really promises (30 s polite, hardcoded and never
overridden; zero on any fence; up to a heartbeat of notice latency first).

And the connect deadline was still spent twice behind an HTTPS proxy — 17.6 s for a
ten-second deadline, 10.5 s once the remainder is recomputed after CONNECT.

A guard that could vanish, and a premise that proved the wrong thing

The multi-address premise check probed one address and accepted 1.5 s — which the kernel's
~3 s ARP give-up passes, while three of those cost ~9 s under the broken implementation,
inside the test's own threshold. It now probes the whole name and requires the timeout to
have been spent once per address. And a skip now fails the run (tests/conftest.py),
because a conditional test that skips has not run.

Round five: three guards this PR claimed were not in the suite

Leading with the failure, because it is about the evidence rather than the code. The
baseline claimed mutation evidence for three tests — multi-address, name lookup, proxy —
that no longer existed. The mutations had really been run; then an edit that replaced a
slice of the test file between two anchors deleted all three while adding two others, and
the claims stayed. For one commit the document described a suite that was not there, and
the helpers those tests used sat in the harness with no callers: replacing the elapsed
deadline with per-address timeouts, removing the bounded lookup, or deleting the
recomputation after CONNECT would every one of them have stayed green.

The three tests are restored, and the claim is now executable: verify_mutations.py
patches each mutation, runs the named test, records whether it reds and on which assertion,
restores, and exits non-zero if any claim is unsupported. Every liveness table in the
baseline is replaced by its output.

python verify_mutations.py  →  12/12 claims verified red against this suite

Two of its own first runs produced meaningless greens and are recorded in the script: a
mutation must be patched onto the path the behaviour would really take, and a patch
target that appears twice is now refused rather than resolved to the first match. The
rule that goes with it: never report mutation evidence for a test that is not in the
committed suite at the moment of reporting.

The save deadline is best-effort, and the documents said otherwise

The twenty seconds was justified from a "fixed 30-second grace". Measured at
origin/master: nothing tells the container how long it has after a stop — not the nine
injected variables (agent/runner.py:2787-2802), not the credentials envelope (its
expires_at is a signature's lifetime), not the job description (timeout_seconds is
documented as "the requested runtime budget, not a deadline",
external/contract.py:485-489), and not the stop object on the heartbeat — inert end to end
and read only by the agent (runners/jobs.py:123-157; StopInstruction: "Nothing reads
this yet."). Thirty is what every call site passes to docker stop today
(agent/runner.py:2817, :2827, :3039) — an observation, not a promise, and the value is
the agent's to choose.

So there is nothing to clamp to, and the justification is withdrawn everywhere it appeared.
What the number buys is a shape, not a guarantee, and the documents now point at exposing
the real remaining deadline as the platform task that would make it exact.

Smaller corrections

The skip guard killed expected xfails (pytest reports them as skipped with wasxfail),
which would have broken this repository's own documented mechanism — excluded and proved
with a probe; its exemption also matched by substring and now compares the test's own name.
The receipt test justified asserting the status word by claiming it "is rendered into what
an operator reads" — it is not, and it now says so. README and AUTHORING promised the
marker and the process cannot disagree; a SIGKILL between the commit and the return makes
that untrue, and they now claim only that the two never differ by decision.

Evidence

python -m pytest -q                                    → 140 passed, 1 skipped
python -m pytest -q --red-for-real                     → 140 passed, 1 skipped
LSPO_ORCHESTRATOR_SRC=… LSPO_ORCHESTRATOR_REF=origin/master python -m pytest -q
                                                       → 141 passed
python verify_mutations.py                             → 12/12 claims verified red

Liveness, four mutations built into a copy of node.py and run against real containers,
each having to fail for the right reason:

Mutation	Result
the SIGTERM handler is never installed	red — receipt test: "its receipt says 'succeeded'"; promptness test: "took 25.2s to go", exit 1
shutdown() put back to close()	red — promptness test only: 25.2 s, exit 20, receipt intact. One line, one test
the marker claims outputs/ghost.csv, never written	red — over-claim test: "the receipt inventories ['outputs/ghost.csv'], which the store never received"
the stop path writes no marker	red — receipt test: "wrote no completion marker"; the other two skip, naming the legal ending
the connect budget removed	red — TLS test: "took 25.2s to go while a TLS handshake was hanging"
the close() override restored	red — mid-body test: "took 24.7s to go while it was reading a body that had stopped arriving"
every marker protected again	red — receipt test: "took 3.2s to go, which is the 3s this test told the store to hold that upload open"
one elapsed deadline replaced by the stdlib's per-address spend, same number	red — multi-address test: "spent 30.6s failing to reach a host with 3 addresses"
the name lookup left unbounded	red — lookup test: "spent 40.6s on a name lookup that was never going to answer"
the check between the receipt landing and exit 0 removed	red — receipt test: "asked to stop and exited 0". This is the mutation that was green last round; making it observable is what the redesign bought
the listener announcing on accept() rather than on evidence	red — harness self-test: "announced a stall before the client had said anything"
the receipt corrected by a second document (round three's design)	red — "the step wrote 2 completion markers for one run"
the receipt's elapsed deadline removed	red — "spent 39.7s on a receipt whose answer was dribbled out over forty"
the connect deadline not recomputed after a proxy's CONNECT	red — "spent 17.6s getting a connection through a proxy that took seven of them to answer"

Documents that said otherwise

AUTHORING.md's skeleton now shows the second half of the handler (with the two measured
facts about close() vs shutdown()), its decision-table row says what that act buys, and
the paragraph claiming reads get the longer timeout "because a stop landing during a read
is noticed as soon as the next block arrives" is corrected — that sentence was the false
claim in prose form. CONFORMANCE.md gains the two lessons about testing a stop: hold the
response open past the grace, or a step that merely waited looks exactly like one that
stopped; and assert the marker in the case where nothing had been produced yet.
CONFORMANCE-BASELINE.md records the measurements, the mutation battery and the moved
counts (128 → 137 passing, conforms_today 41 → 49, basis_reference_quality 24 → 32,
basis_our_policy 23 → 24, basis_contract unchanged at 82). CLAUDE.md and README.md follow. The review round
added the rest: the skeleton sets the flag before the shutdown (it is the version
people paste), says there is no removal on close(), and names the wait that must be
bounded because it cannot be cut; the checklist and the invariants gain the "ask it of
every wait" rule and the success-receipt asymmetry; CONFORMANCE.md gains the two stalls
that need no store.

What this does not prove

That any of it is collected. A local SIGTERM models the node and nothing else: on the
runtime-deadline path the terminal report is refused and the marker is never read, and an
operator's Cancel usually arrives as a SIGKILL. The receipt is written for the runs where
it is read, and for the day those gaps close.

🤖 Generated with Claude Code