Skip to content

ci-build: clean up orphaned per-build upload staging dirs#61

Open
jmandel wants to merge 1 commit into
masterfrom
fix/ci-build-staging-gc
Open

ci-build: clean up orphaned per-build upload staging dirs#61
jmandel wants to merge 1 commit into
masterfrom
fix/ci-build-staging-gc

Conversation

@jmandel

@jmandel jmandel commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Problem

publish / publish-ig receive each build into a staging dir
~/uploading/<RANDOM>[_org_repo] and only mv it into www/ on success. If a
run is interrupted before that final mv — Spot preemption of the build node, a
ci-build pod restart, a failed tar, or an mv error — the staging dir is left
stranded at the root of ~/uploading, and nothing ever reclaims it (the
reindex autoclean only operates inside www/ig).

In production these have accumulated into hundreds of dirs / hundreds of GB on
the ci-build persistent disk (observed ~763 G used, 81% full), dominated by
frequently-built IGs (e.g. many *_HL7_fhir-sdoh-clinicalcare). www/ is the
served site and is fine; everything else at the root of ~/uploading is orphaned
build cruft.

Fix

  • publish / publish-ig — the EXIT trap (which previously only killed the
    keepalive) now also removes the run's own staging dir and any half-rotated .old.
    Guarded with [ -n "$TARGET" ] so an unset variable can never expand to
    rm -rf ~/uploading. It's a no-op on the success path (the dir is already moved).
  • reindex — adds an hourly GC sweep of ~/uploading/* (excluding www/ and
    lost+found) older than GC_AFTER_DAYS (default 2), as a backstop for runs
    that are SIGKILLed past the trap.

Safety

  • The served site (www/) is never touched.
  • -ctime +2 leaves in-flight uploads alone; -ctime matches the conventions
    already used by the reindex autoclean and publish branch-cleanup.
  • The [ -n "$TARGET" ] guard prevents any rm -rf ~/uploading footgun.

Deploy notes

These scripts are baked into the ci-build image, so the fix takes effect after a
rebuild + redeploy. On first run the hourly GC will clear the existing backlog
(everything >2 days old) automatically — no manual cleanup required, though the
disk can be reclaimed sooner with a one-off
find ~/uploading -maxdepth 1 -mindepth 1 -type d ! -name www ! -name 'lost+found' -ctime +2 -exec rm -rf {} \;.

🤖 Generated with Claude Code

publish/publish-ig stage each build under ~/uploading/<RANDOM>[_org_repo] and
only move it into www/ on success. If a run is interrupted before that final mv
(Spot preemption, pod restart, tar/mv failure), the staging dir is stranded at
the root of ~/uploading, and nothing ever reclaims it (the reindex autoclean only
touches www/ig). Over time these orphans accumulate into hundreds of dirs and
hundreds of GB on the ci-build disk.

- publish/publish-ig: EXIT trap now also removes the run's own staging dir (and
  any half-rotated .old), guarded so an unset $TARGET can't expand to ~/uploading.
  No-op on success (already moved).
- reindex: add an hourly GC sweep of ~/uploading/* (excluding www/ and lost+found)
  older than GC_AFTER_DAYS (default 2), as a backstop for runs SIGKILLed past the
  trap. www/ (the served site) is never touched; ctime+2 protects in-flight uploads.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant