tests: move provided.al2 to the deprecated lint list and stop rerunning a non-rerun-safe sync test - #9190
Merged
Merged
Conversation
…ng a non-rerun-safe sync test Two failures from last night's integration runs (32094973517, 32122795247). The pytest 9.1 unblock landed cleanly -- sync-code passes, other-and-e2e went from 135 errors to 0 -- and these are the unrelated leftovers. 1. test_lint_supported_runtimes (failed in both runs) cfn-lint marks provided.al2 EOL as of 2026-07-31, so `sam validate --lint` now exits 1 while the test asserts 0: W2531 Runtime 'provided.al2' was deprecated on '2026-07-31'. Moved it out of supported_runtimes and into test_lint_deprecated_runtimes, which derives the expected code (W2531/E2531/E2533) from cfn-lint's LmbdRuntimeLifecycle.json by date and so does not need touching again as the runtime moves through create-block and update-block. Same treatment as ruby3.2 in #8883 and nodejs20.x in #8947, except those were deleted rather than moved. Checked the rest of the list: only provided.al2 is flagged; java8.al2 is still clean and is the likely next one. 2. TestSyncAdlCasesWithCodeParameter.test_sync_code_function_without_dependencies Reruns cannot help this test and actively hide why it failed. It edits test data in place (update_file writes over code/before/python_function_no_deps/app.py) and the class-scoped fixture that deploys the stack does not re-run, so once attempt 1 has mutated the tree the retries run against inconsistent state. That is what the reported error was: the function came back "Unable to import module 'app': No module named 'numpy'", so _get_lambda_response returned None (error payloads have no "body") and the failure surfaced as `TypeError: the JSON object must be str, bytes or bytearray, not NoneType` -- three wasted deploy cycles and none of the original cause. @pytest.mark.flaky(reruns=0) makes it report the real first-attempt failure immediately. Transient invoke flakes are still covered: _get_lambda_response retries internally. Worth flagging separately: 26 update_file calls across test_sync_adl.py (6), test_sync_code.py (2) and test_sync_watch.py (20) write into code/before, so the same rerun-unsafety applies wherever one of those fails after mutating. Only the test that actually failed is marked here; making them rerun-safe is a bigger change. Testing: with SAM_CLI_DEV=1 (as CI sets), test_lint_supported_runtimes and all 7 test_lint_deprecated_runtimes cases pass -- 8 passed. Verified reruns=0 is honoured (1 attempt versus 4 for an unmarked test). tests/integration/{sync,validate} collect 153 tests. black and ruff clean.
ckawl
approved these changes
Aug 18, 2026
valerena
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue(s) does this change fix?
N/A
Why is this change necessary?
Two failures from last night's integration runs (32094973517, 32122795247).
For context, the pytest 9.1 unblock (#9189) landed cleanly —
sync-codepasses again, andother-and-e2ewent from 135 errors → 0 (all 81 nested-stack logs tests now run). These two are the unrelated leftovers.1.
test_lint_supported_runtimes— fails in both runs, systematiccfn-lint marks
provided.al2EOL as of 2026-07-31, sosam validate --lintnow exits 1 while the test asserts 0:A time bomb rather than a flake — it began failing after that date and fails every run until fixed.
2.
TestSyncAdlCasesWithCodeParameter.test_sync_code_function_without_dependencies— sync-watchReruns cannot help this test and actively hide why it failed. It edits test data in place (
update_filewrites overcode/before/python_function_no_deps/app.py) and the class-scoped fixture that deploys the stack does not re-run, so once attempt 1 has mutated the tree the retries run against inconsistent state.That is exactly what the reported error was. The function came back:
so
_get_lambda_responsereturnedNone(error payloads have nobody) and the failure surfaced asTypeError: the JSON object must be str, bytes or bytearray, not NoneType— three wasted deploy cycles and none of the original cause. Pre-bump this job had 0 reruns and the test passed.How does it address the issue?
provided.al2moves out ofsupported_runtimesand intotest_lint_deprecated_runtimes, which derives the expected code (W2531/E2531/E2533) from cfn-lint'sLmbdRuntimeLifecycle.jsonby date — so it needs no further edits as the runtime moves through create-block and update-block. Same treatment asruby3.2in #8883 andnodejs20.xin #8947, except those were deleted rather than moved.I checked the rest of the list: only
provided.al2is flagged today.java8.al2is still clean and is the likely next one.The sync test gets
@pytest.mark.flaky(reruns=0), so it reports the real first-attempt failure immediately instead of a misleading artifact three deploys later. Transient invoke flakes are still covered —_get_lambda_responseretries internally.What side effects does this change have?
reruns=0means a genuine pre-mutation flake in that one test now fails the build rather than being retried. That is the intended trade: the test cannot pass on a rerun anyway, so the retries only cost time and obscure the cause.Worth flagging for a follow-up: 26
update_filecalls write intocode/before—test_sync_adl.py(6),test_sync_code.py(2),test_sync_watch.py(20) — so the same rerun-unsafety applies wherever one of those fails after mutating. Only the test that actually failed is marked here; making them rerun-safe (restoring the tree, or re-running the infra sync) is a larger change and belongs on its own.Testing
SAM_CLI_DEV=1(as CI sets),test_lint_supported_runtimesand all 7test_lint_deprecated_runtimescases pass — 8 passed. Note a local run withoutSAM_CLI_DEVtests the installedsam, whose older bundled cfn-lint does not flagprovided.al2reruns=0is honoured: 1 attempt versus 4 for an unmarked testtests/integration/{sync,validate}collect 153 tests;blackandruffcleanMandatory Checklist
PRs will only be reviewed after checklist is complete