fix(infra): scope pm2 stop/delete to the staging ecosystem, never all (#912)

[frankbria]

Closes #912.

Problem

Four independent staging paths ran pm2 stop all followed by pm2 delete all, then restarted only ecosystem.staging.config.js.

Staging and production deploy to the same host — both deploy.yml jobs ssh to secrets.HOST and drive pm2 — so an unattended staging health-check remediation stopped and deleted the production backend and frontend, plus any unrelated app on the shared VPS, and never restarted them. A silent production outage lasting until the next production deploy.

Fix

Every stop/delete now targets ecosystem.staging.config.js. pm2 accepts an ecosystem file directly — stop <id|name|namespace|all|json...> — so there is no name list to duplicate and drift across three scripts and a systemd unit:

File	Was
scripts/health-check.sh	pm2 stop all / pm2 delete all — the unattended path that caused the outage
scripts/start-staging.sh	pm2 stop all / pm2 delete all
scripts/deploy-staging.sh	twice: at :87 and :193
systemd/codeframe-staging.service	ExecStop ×2 and ExecReload

The help text those scripts print also told operators to run pm2 stop all / pm2 restart all / pm2 delete all. That teaches the same footgun by proxy, so it now shows the scoped commands.

deploy.yml was already fixed in #1002 (delete by name, not all), so this covers the shell/systemd half the issue names.

Verified against a real pm2 daemon

pm2 6.0.13, three apps, a stand-in production process:

before:   codeframe-production-backend: online
          codeframe-staging-backend:    online
          codeframe-staging-frontend:   online

pm2 stop/delete ecosystem.staging.config.js      <- the new behaviour
after:    codeframe-production-backend: online    ✅ survives

pm2 stop/delete all                              <- the old behaviour
after:    nothing — production was deleted too    ❌

That is acceptance criterion 3 ("a staging restart leaves production pm2 processes online") demonstrated both ways.

Tests

tests/test_pm2_scoping_912.py, beside the existing test_deploy_config.py deploy guards — 4 tests:

• The criterion as an executable scan: no operational pm2 stop|delete|restart all in scripts/ or systemd/. Comment lines are excluded, so fix(deploy): unblock the Deploy workflow (known-hosts, Tailscale routing, pm2 env) #1002's deliberate NOT \pm2 delete all`` warnings do not trip it.
• The scan actually reads the expected files, so it cannot pass by finding nothing.
• The detector matches the original bug — a regression guard whose pattern does not match the thing it guards is worthless.
• Stop paths still name the staging ecosystem, so "not all" cannot be satisfied by removing the stop entirely.

Mutation-checked: restoring pm2 stop all in health-check.sh fails the scan.

Deliberately not changed

grep also finds pm2 stop all in claudedocs/sprint4-*.md and legacydocs/STAGING_SERVER.md. Those are historical sprint records and explicitly legacy documentation, not live runbooks — editing them rewrites a record rather than fixing a path. The issue scopes the criterion to scripts/ and systemd/. Flagging it here so it reads as a decision rather than an oversight.

[github-actions]

Claude finished @frankbria's task in 4m 35s —— View job

---

Bug-hunting review of PR #1006 (pm2 scoping) — ✅ no defects found

GLM review: no defects found. No inline comments — nothing survived a concrete failure-scenario check.

Verified against the checked-out branch (not the diff alone):

• Gather PR context (diff, comments, files)
• Read changed scripts/systemd + test in full
• Analyze for concrete defects (logic, paths, security, data loss)
• Post summary

Scoping is correct. ecosystem.staging.config.js defines only codeframe-staging-backend and codeframe-staging-frontend; pm2 stop/delete/restart <file> matches by those app names, so production (codeframe-production-*) and any unrelated app are untouched. This is the json form of pm2's stop <id|name|namespace|all|json…>, and the author hand-verified it against a live daemon.

No working-directory bug. deploy-staging.sh (L17) and start-staging.sh (L18) both cd "$PROJECT_ROOT", and the bare ecosystem.staging.config.js sits exactly where the pre-existing pm2 start ecosystem.staging.config.js already runs — no intervening cd before the stop/delete at L87/L193 and L79. health-check.sh (PROJECT_ROOT set at L17) uses the absolute $PROJECT_ROOT/… form. Both forms resolve.

systemd unit parses correctly. ExecStop/ExecReload keep the /usr/bin/pm2 absolute executable and add the absolute file path — no quoting or path-resolution issue.

No remaining operational pm2 … all. A read-only scan of scripts/*.sh and systemd/*.service finds none. The regression test's regex (pm2\s+(stop|delete|restart)\s+all\b) matches the original bug, excludes # comment lines, and its file-discovery can't pass by finding nothing (the >= 3 + named-file guards). Checked for false-positive/false-negative paths — none.

No new failure mode. The ecosystem file must already parse for the apps to be running, so the new parse-dependency on stop can't realistically regress a running deploy; all shell calls remain || true-guarded.

Scope note: style, architecture, naming, and coverage belong to the separate general review and are intentionally not covered here.

[github-actions]

Claude finished @frankbria's task in 3m 50s —— View job

---

Bug-hunting review of PR #1006 (pm2 scoping) — synchronize ✅

Precision defect review (logic, security, data-loss, runtime contract). Style/architecture/naming/coverage are covered by the separate general review.

• Gather PR context (diff, comments, files)
• Read changed scripts/systemd + both ecosystem configs + test in full
• Analyze for concrete defects + state a failure scenario for each
• Post summary

No defects found. Full verdict posted as a separate summary comment — scoping is correct (staging ecosystem defines only the two staging app names; production untouched), path resolution is sound across all three scripts and the systemd unit (the synchronize commit's absolute $PROJECT_ROOT/… form closes the only theoretical risk), and no operational pm2 … all remains in any of the 11 scanned files.

[github-actions]

Bug-hunting review of PR #1006 (pm2 scoping) — synchronize — ✅ no defects found

GLM review: no defects found. No inline comments — nothing survived a concrete failure-scenario check.

Reviewed the current branch state (both commits: 4fc5888 + synchronize 5832b2c) against the checked-out repo, not the diff alone. The synchronize commit strengthened the fix by making the stop/delete paths absolute ($PROJECT_ROOT/…), closing the only theoretical risk the first pass flagged.

• Gather PR context (diff, comments, files)
• Read changed scripts/systemd + both ecosystem configs + test in full
• Analyze for concrete defects (logic, paths, security, data loss)
• Post summary

Scoping is correct. ecosystem.staging.config.js (read in full) defines exactly codeframe-staging-backend + codeframe-staging-frontend; ecosystem.production.config.js defines codeframe-production-*. pm2 stop/delete/restart <file> matches by the app names in the file, so production and any unrelated app are untouched — and name-matching is independent of how/where the process was originally started, so the new form still finds a staging app even if it was started from a different path.

No working-directory bug (verified incl. the new absolute paths). All three scripts compute PROJECT_ROOT as an absolute path before the first stop/delete (deploy-staging.sh:16, start-staging.sh:17, health-check.sh:17). The $PROJECT_ROOT/ecosystem.staging.config.js argument therefore resolves regardless of intervening cd — e.g. deploy-staging.sh does cd web-ui/cd .. (L136–153) before its second stop/delete at L193, but the absolute variable is unaffected. The pre-existing bare pm2 start ecosystem.staging.config.js (L199 / start-staging.sh:87) also resolves, because cwd is $PROJECT_ROOT at both points and is not || true-guarded, so a future cwd drift would fail loudly under set -e, not silently.

systemd unit parses and scopes correctly. ExecStop/ExecReload keep the /usr/bin/pm2 absolute executable and pass the absolute /home/frankbria/projects/codeframe/ecosystem.staging.config.js (no spaces → no quoting issue). It matches staging app names, so systemctl stop/reload codeframe-staging leaves production online.

No remaining operational pm2 … all. A read-only scan of all 11 scripts/*.sh + systemd/*.service files finds none. The regression test's regex matches the original bug, excludes #-comment lines, and its file-discovery can't pass by finding nothing (>= 3 files + named-file guards). The "must name the ecosystem file" test prevents satisfying "not all" by simply deleting the stop. Test logic is sound; REPO_ROOT = parents[1] resolves to the repo root.

No new failure mode. Every pm2 stop/delete <file> is || true-guarded, so a missing/unparseable file is swallowed exactly as before — and the ecosystem file must already parse for the apps to be running. No data-loss, race, or resource-leak introduced.

Scope note: style, architecture, naming, and coverage belong to the separate general review and are intentionally not covered here.

View job run

[frankbria]

Phase 11 — demo verification (hard gate)

Every acceptance criterion mapped to outcome evidence, against a real pm2 6.0.13 daemon.

3. A staging restart leaves production pm2 processes online

Ran the exact command the scripts now run, from a different cwd, with an unrelated tenant app alongside the stand-in production process:

before:  codeframe-production-backend: online
         some-other-tenant-app:        online
         codeframe-staging-backend:    online
         codeframe-staging-frontend:   online

$ pm2 stop   "$PROJECT_ROOT/ecosystem.staging.config.js"
$ pm2 delete "$PROJECT_ROOT/ecosystem.staging.config.js"

after:   codeframe-production-backend: online   ✅
         some-other-tenant-app:        online   ✅

And the old behaviour, for contrast:

$ pm2 stop all && pm2 delete all
after:   nothing — production and the tenant app were deleted too   ❌

Edge case checked: apps started by name rather than from the ecosystem file are still matched — pm2 resolves the file to its declared app names, so health-check.sh's unattended remediation works regardless of how the processes were started. Verified: staging pair started via --name, then removed via the ecosystem file, production untouched.

1. Every stop/delete names the staging processes

8 shell call sites + 3 systemd Exec lines, all naming ecosystem.staging.config.js. No pm2 stop|delete remains that does not.

2. The issue's grep returns no operational lines

$ grep -rn 'pm2 \(stop\|delete\) all' scripts/ systemd/
(no output)

Hardening found while demoing

Two scripts used a bare relative ecosystem.staging.config.js. All three cd "$PROJECT_ROOT" at the top so it resolved — but 2>/dev/null || true means a future unbalanced cd would make the stop silently do nothing while the script reported success. Now absolute (5832b2c), and the demo above was re-run from a different cwd to prove it.

PROJECT_ROOT is set at line ~16 in all three scripts, ~60–90 lines before first use, all under set -e. If it were ever empty the path becomes /ecosystem.staging.config.js → not found → swallowed → stops nothing, which fails safe rather than dangerous.

Docs sync (Phase 12b)

No live runbook mentions pm2 stop/delete/restart — deploy/README.md, README.md and docs/ are all clean. The only occurrences are claudedocs/sprint4-* (historical sprint records) and legacydocs/STAGING_SERVER.md, left unchanged as noted in the PR body. No docs update required.

[frankbria]

Cross-family review — codex (Phase 10-pre)

No concrete bugs found in the diff.

The staging scripts resolve PROJECT_ROOT, run from the repo root, and now scope PM2 stop/delete to ecosystem.staging.config.js. The systemd unit uses absolute paths, which avoids cwd ambiguity. The new regression test catches operational pm2 stop/delete/restart all in the staging scripts and service.

APPROVE

Reviewer chain note, for the record: opencode is the documented primary and produced no output on two attempts (~15 min each, killed at timeout) — the same backend failure seen earlier today. codex is the sanctioned fallback; its first attempt also hit the timeout, and a second run with a tighter prompt returned the review above. Recording this because "the reviewer approved" and "the reviewer never ran" look identical from a green PR, and only one of those is a real gate.

Independent passes on this PR: codex (APPROVE) + GLM bot (no defects found), both against the checked-out branch rather than the diff alone.