ci-build: clean up orphaned per-build upload staging dirs#61
Open
jmandel wants to merge 1 commit into
Open
Conversation
publish/publish-ig stage each build under ~/uploading/<RANDOM>[_org_repo] and only move it into www/ on success. If a run is interrupted before that final mv (Spot preemption, pod restart, tar/mv failure), the staging dir is stranded at the root of ~/uploading, and nothing ever reclaims it (the reindex autoclean only touches www/ig). Over time these orphans accumulate into hundreds of dirs and hundreds of GB on the ci-build disk. - publish/publish-ig: EXIT trap now also removes the run's own staging dir (and any half-rotated .old), guarded so an unset $TARGET can't expand to ~/uploading. No-op on success (already moved). - reindex: add an hourly GC sweep of ~/uploading/* (excluding www/ and lost+found) older than GC_AFTER_DAYS (default 2), as a backstop for runs SIGKILLed past the trap. www/ (the served site) is never touched; ctime+2 protects in-flight uploads. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
publish/publish-igreceive each build into a staging dir~/uploading/<RANDOM>[_org_repo]and onlymvit intowww/on success. If arun is interrupted before that final
mv— Spot preemption of the build node, aci-build pod restart, a failed
tar, or anmverror — the staging dir is leftstranded at the root of
~/uploading, and nothing ever reclaims it (thereindexautoclean only operates insidewww/ig).In production these have accumulated into hundreds of dirs / hundreds of GB on
the ci-build persistent disk (observed ~763 G used, 81% full), dominated by
frequently-built IGs (e.g. many
*_HL7_fhir-sdoh-clinicalcare).www/is theserved site and is fine; everything else at the root of
~/uploadingis orphanedbuild cruft.
Fix
publish/publish-ig— theEXITtrap (which previously only killed thekeepalive) now also removes the run's own staging dir and any half-rotated
.old.Guarded with
[ -n "$TARGET" ]so an unset variable can never expand torm -rf ~/uploading. It's a no-op on the success path (the dir is already moved).reindex— adds an hourly GC sweep of~/uploading/*(excludingwww/andlost+found) older thanGC_AFTER_DAYS(default 2), as a backstop for runsthat are
SIGKILLed past the trap.Safety
www/) is never touched.-ctime +2leaves in-flight uploads alone;-ctimematches the conventionsalready used by the
reindexautoclean andpublishbranch-cleanup.[ -n "$TARGET" ]guard prevents anyrm -rf ~/uploadingfootgun.Deploy notes
These scripts are baked into the
ci-buildimage, so the fix takes effect after arebuild + redeploy. On first run the hourly GC will clear the existing backlog
(everything >2 days old) automatically — no manual cleanup required, though the
disk can be reclaimed sooner with a one-off
find ~/uploading -maxdepth 1 -mindepth 1 -type d ! -name www ! -name 'lost+found' -ctime +2 -exec rm -rf {} \;.🤖 Generated with Claude Code