fix: cleanup no longer counts already-destroyed resources as removable - #521
Merged
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
scttfrdmn
force-pushed
the
fix/516-cleanup-phantom-resources
branch
from
August 19, 2026 05:15
243ec54 to
df901b0
Compare
Two compounding defects: 1. enrichInstanceState's NotFound fallback only fires on a batch DescribeInstances error, but EC2 answers an aged-out instance id with an empty result, not an error -- so State stayed "" for exactly the population cleanup exists to sweep. An id absent from a successful response is now marked "deleted" directly, in the success path, instead of relying on an error EC2 doesn't raise for this case. 2. cmd/cleanup.go's removable/running/address split never consulted State at all, so even a resource correctly resolved to "deleted" still landed in removable and in the "N resource(s) would be removed" count -- contradicting its own displayed state one line away. Extracted the split into splitCleanupResources with a new alreadyGone bucket, reported separately as tag-mapping residue and excluded from the removable count. Also added ignoreNotFound tolerance to RemoveResource's instance branch, matching its volume/key-pair/security-group siblings, so a real sweep of stale residue is a satisfied request rather than a doomed Terminate failure. Fixes #516
scttfrdmn
force-pushed
the
fix/516-cleanup-phantom-resources
branch
from
August 19, 2026 05:26
df901b0 to
ce28436
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
spawn cleanup --dry-runreported 9 resources as "would be removed" in an account where all nine had already been destroyed 12+ hours earlier. Two compounding defects, per the issue's own root-cause analysis:enrichInstanceState's NotFound fallback (pkg/aws/cleanup.go) only fires on a batchDescribeInstanceserror. EC2 distinguishes two absence shapes: a syntactically-valid-but-never-existed id fails the whole batch withInvalidInstanceID.NotFound(the fallback's actual trigger), but a real id that has since aged out of the API answers with an empty result — no error at all. SoStatestayed""for exactly the populationcleanupexists to sweep, and the per-id fallback (whose own comment promises a gone instance is "never mistaken for one whose state is merely unknown") never ran.cmd/cleanup.go's removable/running/address split never consultedStateat all — so even a resource the pipeline correctly resolved to"deleted"(volumes, which do 400 on an already-deleted id — the working control in the issue) still landed inremovableand in the "N resource(s) would be removed" headline count, contradicting its own displayeddeletedstate one row away.enrichInstanceStateWith(extracted for testability) now marks any instance id absent from a successfulDescribeInstancesresponse"deleted"directly, in the success path — no longer dependent on an error EC2 doesn't raise for aged-out ids.cmd/cleanup.go's split is extracted intosplitCleanupResources, which adds analreadyGonebucket forState == "deleted", excluded from the removable count and reported separately as tag-mapping residue (the Resource Groups Tagging API's index outlives the resources it describes — that staleness is expected, treating it as ground truth was the bug).RemoveResource's instance branch now toleratesInvalidInstanceID.NotFoundvia the existingignoreNotFoundhelper, matching its volume/key-pair/security-group siblings — a real (non-dry-run) sweep of stale residue is a satisfied request, not a doomedTerminatefailure.Dry run: 0 resource(s) would be removed (N tag mapping(s) are residue for resources that no longer exist).whenremovableis empty but residue exists.Test plan
go build ./...,go vet ./...go test ./pkg/aws/... ./cmd/...TestEnrichInstanceState_AgedOutInstanceIsMarkedDeleted(the core regression — verified it fails against the pre-fix logic:i-aged-out state = "", want deleted),TestEnrichInstanceState_BatchNotFoundStillFallsBack(confirms the existing orphans: unassociated EIPs are reported unconditionally (no spawn/principal attribution), and both remediation hints are wrong — one is destructive #500-era NotFound-batch path is untouched),TestSplitCleanupResources_AlreadyGoneIsNotRemovable(verified it fails against the pre-fix split:removable = [i-live i-gone vol-gone]),TestSplitCleanupResources_NoStateIsStillRemovablegolangci-lint run ./pkg/aws/... ./cmd/...— no new findings in touched filesgofmt -l .— clean[Unreleased]Fixes #516