Skip to content

ci: say why the suite is green — and fix the OOM that saying it uncovered - #4

Merged
makseq merged 2 commits into
mainfrom
docs/ci-comment-no-longer-expected-red
Aug 10, 2026
Merged

ci: say why the suite is green — and fix the OOM that saying it uncovered#4
makseq merged 2 commits into
mainfrom
docs/ci-comment-no-longer-expected-red

Conversation

@makseq

@makseq makseq commented Aug 10, 2026

Copy link
Copy Markdown
Member

Two commits. The first is the comment-only follow-up to #3 that was asked for; the second is a real defect the first one's CI run uncovered, which is why this PR is not comment-only after all.


1. ci: the suite is green because it passes, not because twenty tests are excused

Split out of #3 because the token used there lacked the workflow scope, and GitHub protects files under .github/workflows/ separately.

The header said, in bold, that CI was green because twenty tests ran as strict expected failures, and that fixing one would turn the run red as a prompt to reclassify it. After #3 there is no expected_red_until_fixed test left, so that sentence pointed at machinery no longer carrying anything — and a reader arriving at a red run would have gone looking for a test to promote instead of a bug to fix.

It now says: green because every test passes; the machinery stays, armed and unused, as the tripwire for the next known gap; a failing test is now simply a failure.

The per-basis reporting step piped pytest --collect-only straight to the log, so with nothing to collect it printed no tests collected (129 deselected) three times — which reads like a query that broke rather than an answer of none, and the answer of none is exactly the standing claim the step exists to make. It now prints a count: expected-red on basis_contract: 0. Both that and the --print-labels line are guarded so pytest's exit code 5 cannot fail the step under bash -eo pipefail.

No behaviour change: trigger, runner, Python version, timeout, and the install and run steps are untouched.

2. Release the page cache while streaming

The commit above should have been a no-op. Its CI run came back red, on test_an_input_larger_than_the_container_is_not_fatal — a test that passes locally and passed on main.

It was not flaky. It was a real defect that main had been passing by luck.

node.py streams correctly: it never holds more than one block of an object in memory. That is necessary and not sufficient. A container's memory limit counts the page cache its own reads and writes create, so moving a 128 MiB object through a temporary file leaves the kernel holding 128 MiB of cache for it inside a 64 MiB cgroup. The step's own flat footprint is irrelevant to the OOM killer, and the ending is exit 137 with no marker and no explanation — which classifies as transient, so the platform reads it as "retry me" for work that can never succeed.

Fixed by asking for those pages back every 8 MiB on all four streaming paths — the input download, the multipart upload body, the line count and the digest — with posix_fadvise(POSIX_FADV_DONTNEED), after an fsync on the write side because dirty pages cannot be dropped. Best-effort throughout: it is advice to the kernel, and it costs nothing where it is unavailable.

Measured by moving the harness's own ceiling:

128 MiB input
without this change, into 32 MiB OOM-killed, exit 137
with this change, into 32 MiB passes
with this change, into 64 MiB (the suite's case) passes

The step's code is otherwise identical between those runs. The only difference is who was holding the bytes.

docs/PROTOCOL.md §3.5 and CLAUDE.md now say so, labelled RECOMMENDATION and marked as measured — because "stream and you are safe" is what every author will otherwise conclude from the existing text, and this is the failure that conclusion leads to.


128 passed, 1 skipped locally on this branch.

🤖 Generated with Claude Code

makseq and others added 2 commits August 10, 2026 17:03
…e excused

The header explained that CI was green BECAUSE twenty tests ran as strict expected
failures. That stopped being true when node.py was repaired: there is no
expected_red_until_fixed test left, so a failing test here is now simply a failure —
something to fix in node.py, not something to write down.

The machinery stays, armed and unused, as the tripwire for the next known gap, and
the comment now says that rather than describing a state the repository has left.

The per-basis reporting step is kept and made unambiguous. It printed pytest's own
"no tests collected" line for each basis, which reads like a query that broke; it now
prints a count, because the zero IS the claim being made.

Comments and one reporting line only. The trigger, the runner, the Python version,
the timeout and the install and run steps are untouched, so CI behaviour does not
change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…OOM-killed

CI caught this on the previous commit's branch: test_an_input_larger_than_the
container_is_not_fatal failed on the runner while passing locally and on main.
It was not flakiness in the test. It was a real, latent defect that main had
been passing by luck.

The step streams correctly — it never holds more than one block of an object in
memory. That is not sufficient, because the container's memory limit counts the
PAGE CACHE its own reads and writes create. Moving a 128 MiB object through a
temporary file leaves the kernel holding 128 MiB of cache for it inside a 64 MiB
cgroup, and the program's own flat footprint is irrelevant to the OOM killer.

Fixed by asking for those pages back every 8 MiB, on all four streaming paths:
the input download, the multipart upload body, the line count and the digest.
posix_fadvise(POSIX_FADV_DONTNEED), after an fsync on the write side because
dirty pages cannot be dropped. It is advice rather than a guarantee and costs
nothing where it is unavailable, so it is best-effort throughout.

Measured, by running the harness with the ceiling moved:

  without this change, 128 MiB into 32 MiB   -> OOM-killed, exit 137
  with this change,    128 MiB into 32 MiB   -> passes
  with this change,    128 MiB into 64 MiB   -> passes (the suite's own case)

The step's code is otherwise identical between those runs; the only difference
is who was holding the bytes.

docs/PROTOCOL.md section 3.5 and CLAUDE.md now say this, because "stream and you
are safe" is what every author will otherwise conclude from the existing text,
and the failure it leads to reports itself as exit 137 with no marker and no
explanation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@makseq makseq changed the title ci: the suite is green because it passes, not because twenty tests are excused ci: say why the suite is green — and fix the OOM that saying it uncovered Aug 10, 2026
@makseq
makseq merged commit fa541d2 into main Aug 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant