Skip to content

Pipeline v2: build once, promote the artifact (development branch)#427

Draft
Omicron7 wants to merge 24 commits into
mainfrom
pipeline-v2
Draft

Pipeline v2: build once, promote the artifact (development branch)#427
Omicron7 wants to merge 24 commits into
mainfrom
pipeline-v2

Conversation

@Omicron7

@Omicron7 Omicron7 commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Summary

The v2 pipeline — build-once/promote-the-artifact, per the Agentic Development Lifecycle RFC's Flight phase and the design memo's nine decisions. Long-lived development branch; not for merge until the hoax pilot promotes end-to-end. At release this merges as v2.0.0 with a rolling v2 tag; the v1 surface is never modified (verified: the only v1 file touched is build-dist.yml, whose push trigger gains this branch so dist/ stays built here).

What's here (Passes 1–2)

Actions (provider-agnostic interface, per-runtime modules; CloudRun implemented, ECS/Lambda stubbed):

  • actions/resolve-image — resolve what's running in an environment, or a candidate-<n>/release-<r> tag, to a digest
  • actions/deploy — deploy an explicit digest (hard-fails on tag refs; the invariant is code, not convention), porting v1's CloudRun orchestration (db-migrate first, sidecars preserved, RUNTIME secrets re-attached)
  • actions/dispatch — generic cross-repo workflow_dispatch (replaces trigger-deploy's hardcoded map)

Reusable workflows:

  • build-candidate.yml — env-neutral image from main → shared registry (cru-shared-artifacts/<app>/<app>:candidate-<n> + sha-<gitsha>), router by type, no-change guard reuses existing images for unchanged commits
  • deploy-candidate.yml — candidate tag → release-candidate environment (rc lock)
  • promote.yml — authz (actor must have push on the app repo) → resolve the digest running on rc → deploy to production → tag release-<n> (n = the candidate's build number) → Datadog deployment mark (production lock)
  • rollback.yml — redeploy any release-<r> digest (shared production lock)

Conventions: digests-only deploys; env long names release-candidate/preview map to existing stage/lab nicknames; releases are permanent (never pruned — the registry module's KEEP policy enforces it); telemetry pinned @datadog/datadog-ci@5 with continue-on-error everywhere. Full detail in docs/pipeline-v2.md, including the grants matrix and cru-deploy wrapper sketches.

Validation

76 tests green (vitest), lint clean, 12 action bundles build. Workflow YAML parsed/validated; action input references cross-checked between passes.

Remaining before merge (Pass 3+)

cru-deploy wrappers (+ authz-token secret, WIF vars), hoax Terraform (registry module instance via CruGlobal/cru-terraform-modules#688 + prod env + grants), hoax app workflow pinned @pipeline-v2, end-to-end pilot promote/rollback, then ECS/Lambda passes.

Supersedes #423.

🤖 Generated with Claude Code

Omicron7 and others added 2 commits July 20, 2026 15:46
@Omicron7 @claude
feat: pipeline v2 pass 1 - resolve-image and deploy actions (cloudrun)
391ed2d 
Introduce the build-once/promote-the-artifact pipeline v2 actions:

- actions/resolve-image: resolve a tag or a running environment to a
  digest-pinned image reference in the shared registry
  (cru-shared-artifacts). CloudRun implemented; ecs/lambda stubbed.
- actions/deploy: deploy a pre-built digest reference to a target
  environment, enforcing the digest invariant (tag refs are rejected).
  CloudRun orchestration ported from v1's deploy-cloudrun (db-migrate
  job first, refresh other jobs, rewrite only the app container to
  preserve sidecars, re-attach RUNTIME secrets). ecs/lambda stubbed.

New v2 helpers live in src/v2/ (env mapping, shared-registry paths,
image-ref parsing, Artifact Registry tag->digest resolution via the
REST API + google-auth-library, app-container heuristics). Existing
src/gcp.js helpers are imported read-only; no v1 source is modified.

esbuild emits dist/resolve-image.js and dist/deploy.js; build-dist.yml
now also triggers on the pipeline-v2 branch. Adds vitest coverage for
ref parsing/validation, registry paths, container heuristics,
tag->digest resolution, and deploy orchestration. Docs in
docs/pipeline-v2.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
feat: pipeline v2 pass 2 - candidate/promote/rollback workflows and d…
ebdcad7 
…ispatch action

Add the four reusable workflow_call workflows that wire the Pass 1
resolve-image/deploy actions into the build-once / promote-the-artifact
flow, plus a generic cross-repo dispatch action:

- build-candidate.yml: router (setup + cloudrun/ecs/lambda); cloudrun
  builds candidate-<n> + sha-<gitsha> to the shared registry, with a
  no-change guard that reuses an already-built git sha.
- deploy-candidate.yml: deploy a candidate to release-candidate (no authz).
- promote.yml / rollback.yml: production-locked, authz-gated; promote reuses
  the candidate build number as release-<n>.
- actions/dispatch: generic workflow_dispatch trigger (replaces the hardcoded
  v1 trigger-deploy map); src + esbuild entry + vitest.
- docs/pipeline-v2.md: workflow interfaces, app-info + env translation,
  release numbering, authz contract, locks, grants matrix, Pass 3 sketch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 Omicron7 mentioned this pull request Jul 20, 2026
Closed
Omicron7 and others added 22 commits July 21, 2026 11:18
@Omicron7 @claude
fix: gate BUILD-secrets import behind build-secrets input
de16e53 
The shared BUILD-secrets store (D2) is not provisioned yet: the
cru-shared-artifacts project lacks the Secret Manager API, gcp-secrets
has no per-app filtering for a shared project, and build SAs have no
scoped IAM there. The first hoax candidate build failed on this step.
Gate it behind a new build-secrets input (default false) until the D2
store lands; apps without BUILD secrets are unaffected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
feat(v2): implement ECS pass — resolve/deploy, tag-image, multi-provi…
8edf57a 
…der router

Fill in the ECS stubs for pipeline v2 (build-once/promote-the-artifact) and
restructure the deploy workflows into provider routers.

- src/v2/aws.js: ECR tag<->digest resolution, tags-for-digest, manifest re-tag
  (BatchGetImage -> PutImage), the app-container predicate + compose helper, and
  the service-match regex. All clients use v1's retry config (now policy).
- src/v2/resolve-ecs.js / deploy-ecs.js: ECS keyed on the PROJECT NAME (fixing
  v1's repo-name mismatch). Deploy composes from the FAMILY'S LATEST task-def
  revision (Terraform's template), swaps only the app container's image, refreshes
  RUNTIME secrets, registers a revision, updates every service, and re-points
  EventBridge scheduled tasks; sidecars pass through untouched. assertDigestRef
  enforces the digest invariant. No runtime-project (GCP-only).
- src/v2/image-ref.js: extract provider-neutral ref helpers (parseImageRef etc.)
  so aws.js and gcp.js share them without cross-provider imports; gcp.js re-exports.
- actions/tag-image: provider-agnostic release tagging (cloudrun -> Artifact
  Registry REST addTag; ecs/lambda -> ECR manifest re-tag), replacing promote's
  gcloud CLI step.
- workflows: deploy-candidate/promote/rollback become D9 routers — a lookup job
  (app-info + authz) outputs provider/type/project-ids; gcp/aws jobs gated on
  provider. GCP keeps today's flow (tag-image swaps the gcloud step); AWS assumes
  GitHubDeployECS. build-candidate's build-ecs job is implemented (prod-bound,
  role-suffix GitHubRole). Concurrency locks + authz + datadog conventions kept.
  Lambda stays stubbed.
- docs + tests for every new module.

Terraform follow-ups (aws/ecs/app, separate PR): add <project>-<env>-GitHubRole
for builds (retire TaskRole's GitHub trust); add ecr:PutImage to GitHubDeployECS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
build: rebuild dist/
bfcf2cd 
Rebuild bundles for the v2 ECS pass: new dist/tag-image.js, and dist/deploy.js /
dist/resolve-image.js now include the ECS implementations.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
feat: replace DORA product events with self-owned Datadog events
6cc5e60 
Datadog DORA Metrics is a separate per-committer SKU with unclear
per-app billing exposure; Cru's telemetry cost model is one CI
Visibility committer on cru-deploy. Deploy/promote/rollback now post
structured events to the standard Events API (included with the
platform): service/environment/action/revision tags, promote carries
the candidate, rollback carries rollback:true. A deployments ledger
via an app-info extension is the deferred phase 2 for DORA-style math.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
feat(v2): implement Lambda pass — resolve/deploy, env-neutral candida…
9fc8a79 
…tes, multi-provider aws jobs

Implements the Lambda runtime for pipeline v2, matching the ECS/Cloud Run
contracts:

- resolve-lambda: tag->digest via ECR; environment->digest by reading the
  deployed image off the app's <project>-<nick>* image functions (scratch
  placeholder skipped, mirroring ECS).
- deploy-lambda: v1's ratified selection (Image functions on the app OR scratch
  ECR repo), digest-pinned UpdateFunctionCode, plus v2 hardening — wait for each
  function to finish updating (waitUntilFunctionUpdatedV2) so promote/rollback
  never read back a stale digest. New lambdaWaitForFunctionUpdated helper in
  src/aws.js.
- build-candidate: real build-lambda job mirroring build-ecs, with two
  deliberate differences (--provenance=false so Lambda accepts the manifest; no
  PROJECT_NAME/ENVIRONMENT/BUILD_NUMBER build args — v2 candidates are
  env-neutral).
- deploy-candidate/promote/rollback aws jobs: guard extended to ecs|lambda,
  type-keyed deploy role (GitHubDeployLambda vs GitHubDeployECS), (AWS) job
  names. tag-image already routes lambda down the shared ECR path.
- docs + tests (resolve/deploy lambda, incl. scratch flip, wait, no-match).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
build: rebuild dist/
ff9595e 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
feat(v2): D10 dated release names + idempotent candidate deploys
c5f904c 
Two ratified changes ahead of v2.0.0:

D10 — candidate/release tags carry the BUILD date: candidate-<yyyy-mm-dd>-<n>,
promoted as release-<yyyy-mm-dd>-<n> (full-suffix reuse keeps release ==
candidate identity). Humans get age-at-a-glance in tag listings, dispatch
inputs, and Slack; the build number remains the unique key. Legacy pre-D10
tags stay resolvable everywhere: the reuse guard and promote accept both
forms, and rollback normalizes dated or legacy input (bare build numbers
resolve only for legacy releases).

deploy-candidate is idempotent by default: after resolving the candidate it
reads what release-candidate currently runs and no-ops on a digest match,
making scheduled app workflows (the RFC's nightly-if-changed cadence) safe to
dispatch unconditionally — a quiet night is a true no-op. New force input
covers deliberate same-digest redeploys (e.g. picking up an applied Terraform
task-definition template). Cadence itself stays the app workflow's choice:
per-merge (pilots) or nightly (RFC default) with no reusable-workflow changes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
fix(v2): resolve dated releases from bare build numbers
bd7bcd8 
The first human rollback after D10 typed the natural thing — the bare build
number — and failed at resolve: bare numbers normalized to release-<n>,
which only exists for pre-D10 releases. Fall back in both providers' tag
resolution: when release-<n> misses, search for exactly one matching
release-<yyyy-mm-dd>-<n> and resolve that (0 matches rethrows the original
error; ambiguity never guesses). Failure was pre-mutation by design —
production untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
feat(v2): BUILD secrets are GitHub repo secrets (D2 ratified)
b88adf7 
BUILD-time secrets live in the app repo as Actions secrets named BUILD_<NAME>.
The repo is the isolation boundary -- matching the build's repo-scoped OIDC
identity -- and one mechanism serves all three runtimes, replacing the planned
shared Secret Manager store (API enablement, per-app label filtering, IAM
name-prefix conditions, and a GCP/AWS asymmetry, all now unnecessary).

Each build job exports ONLY BUILD_*-prefixed keys from the inherited secrets
context (prefix stripped: BUILD_NPM_TOKEN -> NPM_TOKEN), no-oping when none
exist; the build-secrets gate input is removed (no current caller passes it).
The prefix filter is load-bearing: secrets:inherit exposes org secrets to the
workflow, and nothing outside BUILD_* may reach build.sh.

Writes need repo admin: DevOps via gh secret set until cru secret gains a
brokered write path through the planned GitHub App. Out of scope for
cru app secrets (runtime-only) -- build secrets are part of the build, which
is GitHub.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
feat(v2): telemetry hardening — DD_VERSION, visible telemetry failure…
22b1f31 
…s, event enrichment, GCP retries

Implements the memo's deferred telemetry hardening plus two flightdeck-adopted
enrichments, all preserving the hard policy that telemetry never fails a deploy.

- DD_VERSION injection: new optional `version` input on the deploy action,
  threaded through the router to all three impls. Cloud Run upserts DD_VERSION on
  the app container's env (services + jobs); ECS upserts into the composed task
  def's app-container `environment` array (composeTaskDefinition stays pure);
  Lambda deliberately ignores it (function env is Terraform-owned — never call
  UpdateFunctionConfiguration). Existing DD_VERSION entries are replaced, never
  duplicated; sidecars untouched. Workflows pass it: deploy-candidate -> the
  candidate tag, promote -> the derived release tag, rollback -> the actual
  release-* tag on the resolved digest (not the bare-number input).

- Visible swallowed-telemetry failures: every Datadog step keeps
  continue-on-error but now appends a `|| echo "::warning ..."` so a failed
  tag/event post annotates the run instead of vanishing.

- Event enrichment (from flightdeck's event shape): the Events API posts gain
  `actor:<github.actor>` and, when the resolved digest carries a sha-<gitsha>
  tag, `sha:<gitsha>` (prefix stripped, omitted cleanly when absent) — the
  deployments-ledger precursor. The datadog-ci pipeline tag steps also gain
  `version:<version>` where known.

- GCP transient-5xx retries: all three Artifact Registry REST client.request
  sites in src/v2/gcp.js get shared gaxios retry options (5 attempts, GET/POST,
  429 + 5xx) so a transient AR 503 no longer fails a resolve, matching the AWS
  SDK clients' maxAttempts:5. The retried tag-create POST is safe (alreadyExists
  is treated as success by the caller).

Tests extended (composeTaskDefinition upsert new/replace/no-op, cloud run env
injection incl. replace, deploy.js router version passthrough, lambda ignores
version, ECS threading, gcp retryConfig passthrough); docs updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
build: rebuild dist/
5397ca5 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
feat(v2): record deploy/promote/rollback events in the deployments le…
3a10536 
…dger

Append each pipeline-v2 deployment event to the CruDeploymentLedger DynamoDB
table (app-info's deployments ledger -- the fleet dashboard / DORA data spine)
directly via `aws dynamodb put-item`, in both provider jobs of deploy-candidate,
promote, and rollback, immediately after the Datadog event step.

- Same telemetry policy as Datadog: continue-on-error + a trailing
  ::warning so a failed write is visible but NEVER fails the deploy.
- GCP (Cloud Run) jobs assume a deploy role (ECS, arbitrarily -- trust is
  cru-deploy-repo-scoped) via configure-aws-credentials placed AFTER all GCP
  steps; AWS jobs already carry the grant on their deploy role.
- The dynamodb:PutItem grant is added inline to the existing deploy roles in
  cru-terraform (no dedicated ledger role); until that applies the steps emit
  benign ::warning annotations and deploys are unaffected.
- docs/pipeline-v2.md: new "Deployments ledger" subsection (schema, writers,
  never-fails policy, GET /deployments read path, v1 deferred to wave-2).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
docs(v2): nightly-if-changed is the onboarding default cadence
edd4e54 
All three pilots run it (staggered noon-UTC crons); per-merge remains
supported per D11.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
fix(v2): DD_VERSION injection is ECS-only — Cloud Run env is Terrafor…
426d95b 
…m-owned

Caught by review question before any deploy ran with it: the Cloud Run
service's ignore_changes covers the app container's IMAGE only; its env is
Terraform-managed (secret refs, environment map, cloud_sql vars). A
pipeline-injected DD_VERSION would show as drift on every plan and be
removed by every apply -- the docker_config disease. Unlike ECS there is no
separate Terraform-owned template for the deploy to compose from, and
ignore_changes cannot exempt a single env entry, so Cloud Run joins Lambda:
version telemetry rides the deployment events. deployCloudRun now accepts
and deliberately ignores version; tests assert the non-injection and that a
pre-existing (Terraform-owned) DD_VERSION passes through untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
feat(v2)!: DD_VERSION is baked at build via --build-arg VERSION
0bec4d6 
Brian's design, replacing deploy-time injection entirely: every candidate
build passes --build-arg VERSION=<yyyy-mm-dd>-<n> (the tag family's bare
suffix -- identical for candidate and future release under D10), and apps
opt in with ARG VERSION + ENV DD_VERSION at the END of the Dockerfile (zero
cache cost, graceful degradation without it).

Strictly better than injection: one true version per build in every
environment (injection made promote look like a version change on identical
bytes); zero Terraform drift on ALL runtimes -- image ENV is invisible to TF
env management, and function-config/service env never sets DD_VERSION, so
the baked value shines through everywhere INCLUDING Lambda (which injection
could never reach); the reused-candidate path keeps its original version,
which is the artifact's identity.

Removes the deploy action's version input, the ECS compose-time upsert, the
router threading, and their tests; events and ledger rows still carry the
full revision tag per deploy (what-was-deployed-where vs what-is-running).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
feat(v2): Slack notifications — rc with ledger-powered compare link, …
61ccf40 
…promote/rollback results

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
feat(v2): rollback-safety classification — advisory EXPAND/CONTRACT v…
1d63d02 
…erdicts on promote

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
build: rebuild dist/
2843550 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
feat(v2)!: run ECS database migrations to completion before service u…
048f1ea 
…pdates

Add a pre-deploy migration phase to deployEcs, mirroring the Cloud Run
db-migrate job. Before any service is touched, DescribeTaskDefinition on the
convention family <project>-<nick>-db-migrate decides whether the app has opted
in (family present) or not (ClientException -> skip). When present, the deploy
composes a revision from the family's latest (release digest + refreshed RUNTIME
secrets, identical to the service path), RunTasks it once (startedBy
cru-pipeline-v2), waits via waitUntilTasksStopped (900s), and requires the
db-migrate container to exit 0. The run configuration (network + launch
type/capacity-provider strategy) is borrowed from the first matching service, or
from the EventBridge scheduled-task target for a jobs-only app; if neither
exists the deploy throws. Any nonzero exit, failure stopCode/stoppedReason, or
wait timeout throws and fails the deploy with the running services untouched.

The matching service list is now fetched once in deployEcs and shared by the
migration phase and updateServices.

This replaces the retired ECS sidecar model, which was verified broken: the app
container's dependsOn on the db-migrate container used condition=START with
essential=false, so the app raced the migration (serving against the un-migrated
schema) and a failed migration never blocked the app or the deploy. Migrations
also re-ran on every task launch/scale-up; the pre-deploy task applies them once
per deploy. The phase runs for every ECS deploy (rc, promote, rollback) since
deployEcs is shared, matching Cloud Run (a rollback's older image no-ops against
already-applied migrations).

New src/aws.js helpers beside the existing ECS ones: ecsDescribeServices,
ecsRunTask, ecsDescribeTasks, ecsWaitUntilTasksStopped (all using RETRY_CONFIG).

BREAKING CHANGE: ECS deploys now run database migrations to completion as a
discrete pre-deploy step keyed on the <project>-<nick>-db-migrate task-definition
family, replacing the sidecar model. Requires the v2 aws/ecs/app module's
top-level database_migrations object; module and workflows adopt as one unit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
build: rebuild dist/
5efebb4 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
fix(v2): don't fail environment resolve on a pre-v2 tag-pinned cloudr…
c17e9c4 
…un image

The no-op guard's 'Check image currently on release-candidate' step resolves
the running image to a shared-registry digest. On an app's FIRST v2 deploy
after v1, the service still runs a tag-pinned image from its old per-project
registry (e.g. bills-stage/container/bills:staging-10108); resolving that tag
against the shared registry threw 'Tag not found', surfacing a red failure
annotation on an otherwise-green run (continue-on-error let the deploy
proceed correctly).

A tag ref whose repo is not the shared registry now returns an empty digest
('no comparable deployment') with an info log instead of throwing. Promote's
environment resolve also improves: a pre-v2 release-candidate now fails at
'Determine release number' with its clearer deploy-a-candidate-first error.

ECS is unaffected (v1/v2 share the app ECR repo, old tags resolve); Lambda
always reports digest refs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Omicron7 @claude
build: rebuild dist/
b9ee436 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Omicron7