From f9ecb45d882a40ec535f7c1110668de2813cee25 Mon Sep 17 00:00:00 2001 From: Tom Wilson Date: Thu, 30 Jul 2026 20:08:40 -0400 Subject: [PATCH] ci: make a smoke failure diagnosable, and stop deploying from a red main MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two holes the 922717f smoke failure exposed. Neither is the crash itself — that was a renderer death on the runner, not reproducible locally (37/37, three runs, same tree). The artifact meant to explain a smoke failure explains nothing. upload-artifact skips anything under a dot-directory unless include-hidden-files is set, and scripts/.smoke is exactly that. The artifact from the failing run was 247 bytes: vite.log, and none of the screenshots or video, despite the job log printing each path as it wrote them. So the one step that exists to diagnose a smoke failure silently discarded 100% of its evidence. Safe to enable: the path stays scoped to scripts/.smoke, the smokes run with VITE_MANAGED_AUTH=false, and the only credentials they inject are apiKey: 'sk-test' and the empty string. Nothing real renders on screen. Render auto-deployed main regardless of CI. The merge gate added in 3b3ecab guards the PR path; nothing guarded the merge itself, so PR #39 shipped managed auth to production while main's smoke job was still red — CI and the deploy racing rather than ordered. Both services now use autoDeployTrigger: checksPass, with the manual-deploy escape hatch noted so a red main cannot strand a hotfix. Deliberately not touched: the renderer crash. Its cause is unproven and tuning deviceScaleFactor or dropping the video would be guessing at it — and might mask it. Fix the evidence pipeline first; let the next occurrence say what actually happened. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01CwNJQ4sAbsG9ZgoioFaSLr --- .github/workflows/check.yml | 10 ++ docs/ci-deploy-gate-prd.html | 203 ++++++++++++++++++++++++++++++ docs/ci-deploy-gate-progress.html | 172 +++++++++++++++++++++++++ render.yaml | 12 ++ 4 files changed, 397 insertions(+) create mode 100644 docs/ci-deploy-gate-prd.html create mode 100644 docs/ci-deploy-gate-progress.html diff --git a/.github/workflows/check.yml b/.github/workflows/check.yml index fc20277..3a0c623 100644 --- a/.github/workflows/check.yml +++ b/.github/workflows/check.yml @@ -137,6 +137,16 @@ jobs: uses: actions/upload-artifact@v4 with: name: smoke-output + # scripts/.smoke is a dot-directory, and this action skips anything + # under one unless told otherwise. Without this the artifact uploaded + # vite.log and silently dropped every screenshot and the video — 247 + # bytes claiming to explain a failure. That is how the renderer crash + # on 922717f ended up undiagnosable. + # + # Safe to enable here because the path stays scoped to scripts/.smoke: + # the smokes run with VITE_MANAGED_AUTH=false and inject only + # apiKey: 'sk-test', so no real credential is ever on screen. + include-hidden-files: true path: | scripts/.smoke/** /tmp/vite.log diff --git a/docs/ci-deploy-gate-prd.html b/docs/ci-deploy-gate-prd.html new file mode 100644 index 0000000..64c944a --- /dev/null +++ b/docs/ci-deploy-gate-prd.html @@ -0,0 +1,203 @@ + + + + + +PRD — CI diagnostics and the deploy gate + + + +
+ +

CI diagnostics and the deploy gate

+

Task slug ci-deploy-gate · branch ci-diagnostics-and-deploy-gate · 2026-07-30 · follow-up to the smoke failure on merge commit 922717f

+ +

Definition

+

In this repo, "the merge gate" means CI that can both block a bad change and +explain a failure after the fact. The gate added in 3b3ecab delivers +the first half only. This task closes the two holes that the 922717f smoke failure +exposed — one in each half.

+ +

What happened, and what it revealed

+

The merge of PR #39 turned main red: the smoke (real browser) job died +with page.waitForTimeout: Page crashed after every assertion had already passed. A +re-run went green, and the same tree passes locally 37/37 across three consecutive runs — so the +crash is environment-specific to the runner, not a code regression.

+

Two structural problems surfaced, neither of which is the crash itself:

+ + + + + + + + + + + + +
#HoleConsequence
AThe failure artifact silently discards every screenshot and the video.A smoke failure is undiagnosable. The artifact from 922717f was 247 bytes containing only vite.log.
BRender auto-deploys main on commit, independent of CI.This deploy went to production while the gate was red. The gate protects the PR path but nothing downstream of the merge.
+ +

Root causes (verified, not assumed)

+ +

A — actions/upload-artifact@v4 excludes hidden paths by default

+

From the action's own documentation: "By default, hidden files are ignored by this action to +avoid unintentionally uploading sensitive information," where hidden means "any file +beginning with . or files within directories beginning with .." The +include-hidden-files input defaults to false.

+

The workflow uploads scripts/.smoke/** — a dot-directory, therefore entirely +filtered out — alongside /tmp/vite.log, which is not hidden. That predicts exactly the +artifact we received: vite.log and nothing else. The screenshots demonstrably existed; +the job log printed each path as it was written.

+ +

B — Render's blueprint defaults to deploy-on-commit

+

Neither service in render.yaml sets a deploy trigger, so both take the default +(deploy on every commit to the linked branch). Render's blueprint spec supports +autoDeployTrigger with three values — commit, checksPass +("Trigger a deploy only if the linked branch's CI checks pass"), and off. The field +supersedes the deprecated autoDeploy boolean.

+ +

Goals

+
    +
  • A failing smoke job uploads its screenshots and video, so the next failure is diagnosable.
    • +
  • Production does not deploy from a red main.
    • +
  • Both changes are declarative in files already in the repo — no new dashboard-only state, no new secrets.
    • +
+ +

Non-goals

+
    +
  • Chasing the renderer crash. Its cause is unproven and not reproducible locally. Tuning deviceScaleFactor or disabling video would be guessing, and could mask the crash rather than fix it. The correct sequence is to fix the evidence pipeline first and let the next occurrence speak.
    • +
  • Retry/rerun logic for flaky jobs — that hides signal, and one occurrence is not a pattern.
    • +
  • Changing what the smoke scripts assert.
    • +
+ +

Constraints

+ + + + + + + + + + + + + + + + + + + +
ConstraintConsequence
Enabling hidden-file upload has a security warning attachedThe action advises validating contents first. Audited: the smokes inject only apiKey: 'sk-test' and apiKey: '', and CI runs with VITE_MANAGED_AUTH=false, so no Clerk session and no real key ever renders. 46 files / ~10 MB, 7-day retention. The path stays scoped to scripts/.smoke/** — it does not broaden to a bare **.
checksPass makes CI a hard dependency of deployingA red or stuck main now blocks production deploys, including hotfixes. That is the intent, but it needs a documented escape hatch (manual deploy from the Render dashboard) or it becomes a footgun during an incident.
Checks must actually report on maincheck.yml runs on: push: branches: [main], so every commit to main produces checks. If that ever changed, checksPass could stall deploys indefinitely.
Blueprint changes need a syncrender.yaml edits take effect when Render syncs the blueprint. The setting cannot be verified from this repo alone — dashboard confirmation is required and is called out as such rather than claimed.
No subagentsSession policy; planner and judge roles performed inline against the same rubric.
+ +

Plan

+ +
+

Step 1 — make the failure artifact carry its evidence

+

Add include-hidden-files: true to the smoke job's upload step, with a comment naming +why it is needed (otherwise the next person deletes it as noise).

+Acceptance criteria +
    +
  • AC1.1 The upload step sets include-hidden-files: true.
    • +
  • AC1.2 The path: list is unchanged — still scoped to scripts/.smoke/** and /tmp/vite.log, not broadened.
    • +
  • AC1.3 A comment explains the hidden-directory rule, so the line survives future edits.
    • +
  • AC1.4 Contents audited for secrets, with the finding recorded (done: fake keys only).
    • +
  • AC1.5 The workflow remains valid YAML and CI runs green on the PR.
    • +
+

Evidence: diff; YAML parse; a green CI run on the PR.

+
+ +
+

Step 2 — gate the production deploy on CI

+

Set autoDeployTrigger: checksPass on both services in render.yaml, with a +comment recording why and how to override during an incident.

+Acceptance criteria +
    +
  • AC2.1 Both build and hyper-build-api declare autoDeployTrigger: checksPass.
    • +
  • AC2.2 The value is one of the three the spec allows; the deprecated autoDeploy boolean is not used.
    • +
  • AC2.3 A comment documents the escape hatch — a manual deploy from the Render dashboard — so a red main cannot strand a hotfix.
    • +
  • AC2.4 render.yaml remains valid YAML.
    • +
  • AC2.5 The limits of verification are stated honestly: taking effect requires a Render blueprint sync, which cannot be confirmed from the repo.
    • +
+

Evidence: diff; YAML parse; spec citation.

+
+ +
+

Step 3 — validate and ship

+Acceptance criteria +
    +
  • AC3.1 Both files parse as YAML.
    • +
  • AC3.2 The full suite is unaffected — 810 tests still green (this task touches no application code).
    • +
  • AC3.3 CI passes on the PR, including the smoke job.
    • +
  • AC3.4 No application source file is modified by this task.
    • +
+

Evidence: parse output; suite output; PR check results; git diff --stat.

+
+ +

Validation commands

+
+
node -e "require('yaml')" 2>/dev/null || python3 -c "import yaml,sys; yaml.safe_load(open('.github/workflows/check.yml')); yaml.safe_load(open('render.yaml')); print('both parse')"
+npm test               # unaffected; 197 gleam + 304 vitest
+git diff --stat        # expect only .github/workflows/check.yml, render.yaml, docs/
+
+
+ +

Definition of done

+

The smoke job's failure artifact would contain the screenshots and video; both Render services +declare checksPass; both files parse; CI is green on the PR; no application code +touched. The report states plainly that the Render side is verified as configuration, not +as observed behaviour, and names the dashboard check needed to close that gap.

+ +

Judge rubric

+ + + + + + + + +
AreaJudgment
CorrectnessDo the two changes actually address the diagnosed causes, each verified against primary documentation rather than assumed?
SecurityWas the hidden-file upload audited before enabling, per the action's own warning? Is the path still scoped?
RestraintDoes the change resist "fixing" the unproven crash?
OperabilityDoes the deploy gate come with an escape hatch, or does it strand hotfixes during an incident?
TasteDo the comments explain why in this repo's established voice — the existing render.yaml and check.yml comments are unusually narrative and explain rationale, not mechanics.
HonestyIs the unverifiable part (blueprint sync) labelled as such rather than reported as done?
+

Taste threshold: below 4 triggers a repair loop. This repo's CI and blueprint comments carry real +authorial voice ("The merge gate." / "Until now there was none"), so a bland comment here would be a +regression in fit.

+ +

Risks

+ + + + + + +
RiskHandling
Hidden-file upload leaks a secretAudited before enabling; only fake keys present; path stays scoped; 7-day retention.
checksPass strands a hotfix behind red CIDocumented escape hatch in the comment (manual deploy). Accepted trade: a red gate blocking a deploy is the feature.
Blueprint sync does not apply the fieldExplicitly flagged for dashboard confirmation; not claimed as verified.
The renderer crash recursExpected and acceptable — the point of Step 1 is that the next one arrives with evidence attached.
+ +
+ + diff --git a/docs/ci-deploy-gate-progress.html b/docs/ci-deploy-gate-progress.html new file mode 100644 index 0000000..e8fb0f1 --- /dev/null +++ b/docs/ci-deploy-gate-progress.html @@ -0,0 +1,172 @@ + + + + + +Progress — CI diagnostics and the deploy gate + + + +
+ +

Progress — CI diagnostics and the deploy gate

+

Task slug ci-deploy-gate · branch ci-diagnostics-and-deploy-gate · PRD: ci-deploy-gate-prd.html

+ +

Status: complete — both holes +closed. The artifact fix was verified empirically, by deliberately failing CI once and +inspecting what came back, rather than trusting the documentation.

+ +

Investigation that led here

+ + + + + + +
QuestionAnswerHow
Was the smoke failure a real regression?noRan the smoke locally against the exact merged tree three times: 37/37 each. CI died around check 20.
Is this job known-flaky?no — first occurrenceEvery prior CI failure in the repo was a different job (server, module resolution, 2026-07-28). The smoke job had never failed.
What actually killed it?unprovenChrome renderer crash (page.waitForTimeout: Page crashed). Plausible mechanism: video at deviceScaleFactor: 2 (2880×1800 frames) plus an in-page WebContainer on a 2-core runner. Stated as a hypothesis, not a finding.
Why can't we say more?the artifact was empty247 bytes containing only vite.log. That is the defect this task fixes.
+ +

Root causes, verified against primary sources

+ + + + + + + + + + + + +
CauseCitation
Aupload-artifact@v4 excludes hidden paths by default; include-hidden-files defaults to false. Hidden means any file beginning with . or inside a directory beginning with . — so all of scripts/.smoke/.Action documentation: "By default, hidden files are ignored by this action to avoid unintentionally uploading sensitive information."
BRender's blueprint defaults to deploy-on-commit; neither service declared a trigger.Blueprint spec: autoDeployTrigger accepts commit, checksPass ("Trigger a deploy only if the linked branch's CI checks pass"), off. Supersedes the deprecated autoDeploy boolean.
+

The theory predicted the observation exactly: /tmp/vite.log is not hidden and +survived; everything under the dot-directory did not. That match is what moved this from guess to +diagnosis.

+ +

Step checklist

+ + + + + +
StepStatusCriteria
1 — failure artifact carries its evidencepass5/5
2 — gate the production deploy on CIpass5/5 (one config-verified only — see AC2.5)
3 — validate and shippass4/4
+ +

Step 1 — evidence

+ + + + + + + +
ACVerdictEvidence
1.1 flag setpassParsed the workflow: include-hidden-files: True
1.2 path not broadenedpassParsed value still scripts/.smoke/**\n/tmp/vite.log — scoped, not a bare **
1.3 comment explains whypassNames the dot-directory rule and the 247-byte artifact it caused
1.4 contents auditedpassOnly apiKey: 'sk-test' and apiKey: '' appear in the smokes; CI runs VITE_MANAGED_AUTH=false, so no Clerk session and no real key can render. 46 files / ~10 MB.
1.5 valid YAML + green CIpassParses; CI green on the final branch state
+ +

Empirical proof, not just configuration

+

Asserting the flag is set is not the same as proving the artifact works — the entire lesson of the +preceding session was that a thing can look correct and be wrong. So the fix was tested for real: a +throwaway commit added a TEMP force a failure step immediately after the smoke scripts, +so they would run to completion and then fail, triggering the if: failure() +upload. The resulting artifact was downloaded and inspected, then the commit was dropped from the +branch by reset and force-push, leaving a clean history.

+
+ + + + + + + +
ArtifactBefore — run on 922717fAfter — forced-failure run 30592790179
size247 bytes8,934,307 bytes (~8.9 MB)
files1 — tmp/vite.log30
screenshots026
videos03 (.webm, one per smoke script)
coveragenoneall three smokes: agent-harness/, interview/, layout-modes/
+
+

A screenshot from the recovered artifact was opened and inspected. It shows the app mid-turn +("Working… / Listed 13 files") with no credential anywhere on screen — confirming the +security audit against rendered output, not merely against the source that generates it. It is also +exactly the class of evidence that was missing for 922717f: had this been captured then, +the frame at crash time would have been available.

+ +

Step 2 — evidence

+ + + + + + + +
ACVerdictEvidence
2.1 both services gatedpassParsed: build → checksPass, hyper-build-api → checksPass
2.2 spec-valid, not deprecatedpassautoDeploy absent from both services; value is one of the three the spec allows
2.3 escape hatch documentedpassComment names manual deploy from the Render dashboard, so a red main cannot strand a hotfix
2.4 valid YAMLpassrender.yaml -> parses OK
2.5 limits statedpassVerified as configuration, not as observed behaviour. Taking effect requires a Render blueprint sync; that cannot be confirmed from this repo. Dashboard check required — see follow-ups.
+ +

Step 3 — evidence

+ + + + + + +
ACVerdictEvidence
3.1 both parsepassPython yaml.safe_load on both files
3.2 suite unaffectedpass197 gleam + 304 vitest, unchanged
3.3 CI green on PRpassFinal branch state, after the temp commit was dropped
3.4 no app source touchedpassgit diff --name-only matches nothing under src/ or server/ — 2 config files, +22 lines
+ +

Inline judge review

+

No subagents (session policy); judge role performed inline against the PRD rubric.

+ + + + + + + + +
AreaVerdictNote
CorrectnesspassBoth causes verified against primary docs, then A verified empirically. The predicted-vs-observed match (non-hidden file survived, hidden tree did not) is what makes the diagnosis more than plausible.
SecuritypassAudited before enabling, per the action's own warning. Path stays scoped; only fake credentials exist in the smokes.
RestraintpassThe renderer crash was deliberately left alone. Tuning it would have been guessing, and could mask recurrence.
OperabilitypassEscape hatch documented in the file a reader hits when the gate blocks them.
Taste4/5Comments match the repo's narrative CI voice ("The merge gate. Until now there was none") and explain rationale over mechanics. Not a 5 because the change is inherently two config lines — there is little room for craft beyond saying why.
HonestypassThe Render half is labelled config-verified rather than reported as working; the crash cause is labelled a hypothesis.
+ +

Files changed

+
    +
  • .github/workflows/check.yml — include-hidden-files: true + rationale
    • +
  • render.yaml — autoDeployTrigger: checksPass on both services + escape hatch
    • +
  • docs/ci-deploy-gate-{prd,progress}.html — these artifacts
    • +
+ +

Residual risks and follow-ups

+
    +
  • Render blueprint sync is unconfirmed. Worth checking in the dashboard that both services show "deploy on CI pass" after this merges. Until then the gate is declared, not proven.
    • +
  • The renderer crash is still unexplained, and may recur. That is the accepted outcome — the next occurrence now arrives with screenshots and video attached.
    • +
  • checksPass makes CI a dependency of shipping. A stuck or misconfigured workflow now blocks deploys. The manual-deploy hatch is the mitigation.
    • +
  • Unrelated, still open: F3 from the smoke test — provisioning is not idempotent across an empty users table, leaving orphaned OpenRouter keys. Needs a product decision.
    • +
+ +

Handoff log

+
+
2026-07-30  begin         ci-deploy-gate  step 0  investigate why CI didn't complete on 922717f
+2026-07-30  finding       ci-deploy-gate  step 0  not a regression: 37/37 locally x3; smoke job had
+                                                   never failed before in this repo
+2026-07-30  root-cause    ci-deploy-gate  step 0  A: upload-artifact skips dot-dirs by default
+                                                   B: render defaults to deploy-on-commit
+2026-07-30  plan-approved ci-deploy-gate  step 0  inline plan review; crash-chasing ruled a non-goal
+2026-07-30  step-1-done   ci-deploy-gate  step 1  include-hidden-files: true; contents audited
+2026-07-30  step-2-done   ci-deploy-gate  step 2  autoDeployTrigger: checksPass on both services
+2026-07-30  step-3-done   ci-deploy-gate  step 3  yaml parses; 501 tests unaffected; config-only diff
+
+
+ +
+ + diff --git a/render.yaml b/render.yaml index 698ab09..7158504 100644 --- a/render.yaml +++ b/render.yaml @@ -2,6 +2,14 @@ services: - type: web name: build runtime: static + # Deploy only from a green main. The merge gate added in 3b3ecab guards the + # PR path, but nothing guarded what happens after the merge: PR #39 shipped + # to production while main's smoke job was still red. Auto-deploy defaults + # to every commit, so CI and the deploy were racing rather than ordered. + # + # ESCAPE HATCH: a red or stuck main now blocks deploys, hotfixes included. + # Deploy manually from the Render dashboard when you need to go around it. + autoDeployTrigger: checksPass envVars: - key: GLEAM_VERSION value: 1.16.0 @@ -62,6 +70,10 @@ services: - type: web name: hyper-build-api runtime: node + # Same gate as the static site — see the comment there. It matters more + # here: this service holds the users table and the provisioned keys, so a + # bad deploy costs more than a bad page. + autoDeployTrigger: checksPass # Disks require a paid instance; previews stay off for this service (PR # previews of the frontend point at this production API, which is inert # until VITE_MANAGED_AUTH is on).