Skip to content

tests: move provided.al2 to the deprecated lint list and stop rerunning a non-rerun-safe sync test - #9190

Merged
roger-zhangg merged 1 commit into
developfrom
fix-nightly-integ-eol-and-rerun
Aug 18, 2026
Merged

tests: move provided.al2 to the deprecated lint list and stop rerunning a non-rerun-safe sync test#9190
roger-zhangg merged 1 commit into
developfrom
fix-nightly-integ-eol-and-rerun

Conversation

@roger-zhangg

Copy link
Copy Markdown
Member

Which issue(s) does this change fix?

N/A

Why is this change necessary?

Two failures from last night's integration runs (32094973517, 32122795247).

For context, the pytest 9.1 unblock (#9189) landed cleanly — sync-code passes again, and other-and-e2e went from 135 errors → 0 (all 81 nested-stack logs tests now run). These two are the unrelated leftovers.

1. test_lint_supported_runtimes — fails in both runs, systematic

cfn-lint marks provided.al2 EOL as of 2026-07-31, so sam validate --lint now exits 1 while the test asserts 0:

W2531: Runtime 'provided.al2' was deprecated on '2026-07-31'.
Error: Linting failed. At least one linting rule was matched to the provided template.

A time bomb rather than a flake — it began failing after that date and fails every run until fixed.

2. TestSyncAdlCasesWithCodeParameter.test_sync_code_function_without_dependencies — sync-watch

Reruns cannot help this test and actively hide why it failed. It edits test data in place (update_file writes over code/before/python_function_no_deps/app.py) and the class-scoped fixture that deploys the stack does not re-run, so once attempt 1 has mutated the tree the retries run against inconsistent state.

That is exactly what the reported error was. The function came back:

Unable to import module 'app': No module named 'numpy'

so _get_lambda_response returned None (error payloads have no body) and the failure surfaced as TypeError: the JSON object must be str, bytes or bytearray, not NoneType — three wasted deploy cycles and none of the original cause. Pre-bump this job had 0 reruns and the test passed.

How does it address the issue?

provided.al2 moves out of supported_runtimes and into test_lint_deprecated_runtimes, which derives the expected code (W2531/E2531/E2533) from cfn-lint's LmbdRuntimeLifecycle.json by date — so it needs no further edits as the runtime moves through create-block and update-block. Same treatment as ruby3.2 in #8883 and nodejs20.x in #8947, except those were deleted rather than moved.

I checked the rest of the list: only provided.al2 is flagged today. java8.al2 is still clean and is the likely next one.

The sync test gets @pytest.mark.flaky(reruns=0), so it reports the real first-attempt failure immediately instead of a misleading artifact three deploys later. Transient invoke flakes are still covered — _get_lambda_response retries internally.

What side effects does this change have?

reruns=0 means a genuine pre-mutation flake in that one test now fails the build rather than being retried. That is the intended trade: the test cannot pass on a rerun anyway, so the retries only cost time and obscure the cause.

Worth flagging for a follow-up: 26 update_file calls write into code/beforetest_sync_adl.py (6), test_sync_code.py (2), test_sync_watch.py (20) — so the same rerun-unsafety applies wherever one of those fails after mutating. Only the test that actually failed is marked here; making them rerun-safe (restoring the tree, or re-running the infra sync) is a larger change and belongs on its own.

Testing

  • With SAM_CLI_DEV=1 (as CI sets), test_lint_supported_runtimes and all 7 test_lint_deprecated_runtimes cases pass — 8 passed. Note a local run without SAM_CLI_DEV tests the installed sam, whose older bundled cfn-lint does not flag provided.al2
  • Verified reruns=0 is honoured: 1 attempt versus 4 for an unmarked test
  • tests/integration/{sync,validate} collect 153 tests; black and ruff clean

Mandatory Checklist

PRs will only be reviewed after checklist is complete

…ng a non-rerun-safe sync test

Two failures from last night's integration runs (32094973517, 32122795247). The pytest 9.1
unblock landed cleanly -- sync-code passes, other-and-e2e went from 135 errors to 0 -- and
these are the unrelated leftovers.

1. test_lint_supported_runtimes (failed in both runs)

cfn-lint marks provided.al2 EOL as of 2026-07-31, so `sam validate --lint` now exits 1
while the test asserts 0:

  W2531 Runtime 'provided.al2' was deprecated on '2026-07-31'.

Moved it out of supported_runtimes and into test_lint_deprecated_runtimes, which derives
the expected code (W2531/E2531/E2533) from cfn-lint's LmbdRuntimeLifecycle.json by date and
so does not need touching again as the runtime moves through create-block and update-block.
Same treatment as ruby3.2 in #8883 and nodejs20.x in #8947, except those were deleted
rather than moved. Checked the rest of the list: only provided.al2 is flagged; java8.al2 is
still clean and is the likely next one.

2. TestSyncAdlCasesWithCodeParameter.test_sync_code_function_without_dependencies

Reruns cannot help this test and actively hide why it failed. It edits test data in place
(update_file writes over code/before/python_function_no_deps/app.py) and the class-scoped
fixture that deploys the stack does not re-run, so once attempt 1 has mutated the tree the
retries run against inconsistent state. That is what the reported error was: the function
came back "Unable to import module 'app': No module named 'numpy'", so
_get_lambda_response returned None (error payloads have no "body") and the failure surfaced
as `TypeError: the JSON object must be str, bytes or bytearray, not NoneType` -- three
wasted deploy cycles and none of the original cause.

@pytest.mark.flaky(reruns=0) makes it report the real first-attempt failure immediately.
Transient invoke flakes are still covered: _get_lambda_response retries internally.

Worth flagging separately: 26 update_file calls across test_sync_adl.py (6),
test_sync_code.py (2) and test_sync_watch.py (20) write into code/before, so the same
rerun-unsafety applies wherever one of those fails after mutating. Only the test that
actually failed is marked here; making them rerun-safe is a bigger change.

Testing: with SAM_CLI_DEV=1 (as CI sets), test_lint_supported_runtimes and all 7
test_lint_deprecated_runtimes cases pass -- 8 passed. Verified reruns=0 is honoured (1
attempt versus 4 for an unmarked test). tests/integration/{sync,validate} collect 153
tests. black and ruff clean.
@roger-zhangg
roger-zhangg requested a review from a team as a code owner August 18, 2026 16:35
@roger-zhangg
roger-zhangg enabled auto-merge August 18, 2026 16:58
@roger-zhangg
roger-zhangg added this pull request to the merge queue Aug 18, 2026
Merged via the queue into develop with commit 5b6ebdb Aug 18, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants