ci(longhaul): add image build, deploy, monitor + auto-upgrade workflows#413
Draft
WentingWu666666 wants to merge 5 commits into
Draft
ci(longhaul): add image build, deploy, monitor + auto-upgrade workflows#413WentingWu666666 wants to merge 5 commits into
WentingWu666666 wants to merge 5 commits into
Conversation
|
🤖 Auto-triaged by documentdb-triage-tool. Applied: Reasoningcomponent from path globs (ci); effort from diff stats (672+0 LOC, 5 files); LLM failed: Invalid response body while trying to fetch https://api.anthropic.com/v1/messages: Premature close If a label is wrong, remove it manually and ping |
Introduces the GitHub Actions plumbing for the long-haul test driver that landed in PR documentdb#405. Three workflows + two deploy manifests: * .github/workflows/longhaul-image-build.yml Build/push test/longhaul image to GHCR on main push and on demand. Tags every run as :sha-<short> (immutable) plus :main. * .github/workflows/longhaul-deploy.yml Roll an image onto the long-haul AKS cluster. Auto-triggered after a successful image build (pins to :sha-<short>) and via workflow_dispatch for rollbacks. Uses a namespace-scoped kubeconfig in the LONGHAUL_KUBECONFIG secret. * .github/workflows/longhaul-monitor.yaml Hourly health poll: Deployment ready, report ConfigMap fresh (<=2h), test result != FAIL. Auto-upgrade and DocumentDB version publishing are intentionally left out and will land in a separate upgrade PR. * test/longhaul/deploy/deployment.yaml Single-replica Deployment + ConfigMap. Image fields templated (__OWNER__/__IMAGE_TAG__) for the deploy workflow to substitute. Aligned with the post-PR-405 env-var surface (LONGHAUL_DOCUMENTDB_URI, no NUM_VERIFIERS) and the credential secret name documented in test/longhaul/README.md (longhaul-documentdb-credentials with a uri key). * test/longhaul/deploy/rbac.yaml Namespace-scoped ServiceAccount/Role/RoleBinding (pods, dbs, configmaps) plus a ClusterRole for metrics.k8s.io. Splits PR documentdb#348 part 3 of 5. Operator/DocumentDB auto-upgrade plus post-upgrade verification follow in PR-4. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu <wentingwu@microsoft.com>
WentingWu666666
force-pushed
the
developer/wentingwu/longhaul-cicd
branch
from
8b658e6 to
db58044
Compare
WentingWu666666
changed the title
Self-review found 15 issues across the three workflows + deploy
manifest. Fixed in-place; also added a PR-time lint workflow so
future regressions get caught before merge instead of post-merge
against the live cluster.
Critical:
1. longhaul-image-build: build context was test/longhaul, but the
module's go.mod has `replace` directives for ../shared and
../../operator/src. Set context=. and file=test/longhaul/Dockerfile
so the replace targets are inside the build context.
2. longhaul-monitor: operator version check used
`-l app.kubernetes.io/name=documentdb-operator` + .items[0]; the
chart applies that label to three Deployments (operator, sidecar
injector, wal replica). Target the operator Deployment by name
(env.OPERATOR_DEPLOYMENT=documentdb-operator).
3. longhaul-monitor: missing `packages: read` permission; the GHCR
version lookups (`gh api /orgs/.../packages/...`) silently fall
back to chart defaults without it.
4. longhaul-deploy: header comment claimed kubeconfig is scoped via
rbac.yaml. rbac.yaml is the driver pod's in-cluster SA — orthogonal
to the GHA kubeconfig, which the upgrade job needs cluster-admin
for (helm upgrade in another ns + CRD apply). Rewrote header.
Major:
5. All three workflows: secrets.LONGHAUL_KUBECONFIG was inlined into
`run:` blocks via ${{ secrets.* }}. Switch to env: passthrough so
`$` and backticks in the secret cannot shell-inject.
6. longhaul-monitor: ConfigMap longhaul-versions was applied with
`kubectl create --dry-run | kubectl apply -f -`, which sends a full
body and removes any keys the driver may have added (last-applied,
attempt counters, etc.). Switch to server-side apply with
--field-manager=longhaul-monitor so the monitor co-owns only the
fields it sets.
7. longhaul-deploy: redundant `kubectl set image` + always
rollout-restart after `kubectl apply` which already rolls a pod
when the image changes. Dropped the set-image; gate the restart on
mutable tags only (main / latest / sha-*); pinned semver tags
never need a forced restart.
8. (deferred) workflow_run only fires on default branch — covered by
the new lint-longhaul-workflows.yml which runs on PRs.
9. (verified, not a bug) GHCR documentdb image path matches the
URL-encoded subpackage form used in the operator source
(operator/src/internal/utils/constants.go).
Minor:
10. Pinned runs-on to ubuntu-22.04 across all jobs (was ubuntu-latest
in places) so future runner image roll-forward doesn't surprise us.
11. Added concurrency: groups to all three workflows (cancel-in-progress
true for build/lint, false for deploy/monitor to avoid mid-rollout
cancellation).
12. Added timeout-minutes to every job.
13. longhaul-image-build: replaced
cache-{from,to}=type=registry,ref=...:buildcache (accumulates in
GHCR, no GC) with type=gha so cache cycles automatically.
14. longhaul-monitor: added "Dump driver logs on failure" step that
tails 200 lines from the long-haul pod + describes it whenever any
prior step in check-longhaul fails. Run summaries now contain
enough triage context without a kubectl session.
15. test/longhaul/deploy/deployment.yaml ConfigMap was missing the
tunables LONGHAUL_STEADY_STATE_WAIT, LONGHAUL_REPORT_INTERVAL,
LONGHAUL_RESET_DATA. Added with explicit values (60s, 1h, false)
that match the in-code defaults and commentary about when to
override.
New CI gating:
.github/workflows/lint-longhaul-workflows.yml runs on PRs touching
any of the long-haul workflow files, Dockerfile, deploy manifests, or
the operator chart:
- actionlint (+ embedded shellcheck on run: blocks)
- yamllint (strict)
- kubeconform against k8s 1.30 schemas + a Datree CRD catalog
fallback (skips DocumentDB which we don't ship a CRD schema for)
- helm template smoke against operator/documentdb-helm-chart with a
grep assertion that the rendered output still contains a
Deployment named exactly `documentdb-operator` — protects against
regressions of the fix in (2)
- docker buildx with push:false, mirroring production build exactly
to catch context/replace-directive bugs at PR time
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Wenting Wu <wentingwu@microsoft.com>
- lint-longhaul-workflows.yml: replace deprecated reviewdog input
fail_on_error with fail_level: error.
- Group consecutive $GITHUB_OUTPUT redirects with { ...; } >> file
to satisfy shellcheck SC2129 (image-build, deploy, monitor).
- monitor: bind ready output to a shell variable before the numeric
-lt comparison to satisfy shellcheck SC2170.
- image-build: drop extra spaces after cache-to: colon (yamllint).
- monitor: use two spaces before inline comments (yamllint).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Wenting Wu <wentingwu@microsoft.com>
Build chart dependencies (cloudnative-pg) before templating and pass --api-versions cert-manager.io/v1/Certificate so the cert-manager preflight check passes during offline rendering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu <wentingwu@microsoft.com>
WentingWu666666
changed the title
The 'Publish desired DocumentDB version' step performs a server-side apply on the longhaul-versions ConfigMap with field manager 'longhaul-monitor'. When the ConfigMap already exists and its fields (e.g. .data.last-updated) are owned by another manager such as 'kubectl-client-side-apply' (from an earlier 'kubectl create'/client-side apply), the SSA fails with a field ownership conflict: error: Apply failed with 1 conflict: conflict with "kubectl-client-side-apply" using v1: .data.last-updated Add --force-conflicts so the monitor, which is the intended authoritative writer of desired-documentdb-version and last-updated, cleanly takes ownership of those fields. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Wenting Wu <wentingwu@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on PR #405 (long-haul driver core, merged).
Adds the GitHub Actions plumbing + Kubernetes manifests to build, deploy,
monitor, and auto-upgrade the long-haul test driver on the dedicated
AKS cluster.
What this PR adds
Workflows (
.github/workflows/)longhaul-image-build.ymlmain(paths:test/longhaul/**), workflow_dispatchtest/longhaul/Dockerfile, push to GHCR with:sha-<short>(immutable) +:maintagslonghaul-deploy.yml:sha-<short>), workflow_dispatch (manual rollback)kubectl applyDeployment manifest, set image, wait for rolloutlonghaul-monitor.yamllonghaul-versionsConfigMap; the driver performs the in-band DocumentDB upgrade as a load-aware operation so continuous writers/verifiers can catch any data-integrity regressions.Manifests (
test/longhaul/deploy/)deployment.yaml— single-replica Deployment + tunable ConfigMap; image fields templated (__OWNER__/__IMAGE_TAG__) for the deploy workflow.rbac.yaml— namespace-scoped ServiceAccount/Role/RoleBinding (pods,documentdb.io/dbs, configmaps) + ClusterRole formetrics.k8s.io.Setup required before the workflows can run
Cluster admin one-time bootstrap (the deployer ServiceAccount is namespace-scoped by design):
kubectl apply -f test/longhaul/deploy/setup.yaml(already on main via PR test(longhaul): add long-haul test driver core #405; namespace + DocumentDB CR + credentials placeholder)kubectl apply -f test/longhaul/deploy/rbac.yamllonghaul-documentdb-credentialssecret with keyuri(seetest/longhaul/README.md)longhaul-testServiceAccount, store as repo secretLONGHAUL_KUBECONFIGWhy draft
Want to dry-run the workflows end-to-end on the long-haul AKS cluster before un-drafting.
Test plan
python -c "import yaml; yaml.safe_load(...)"parses all three workflow YAMLs.LONGHAUL_KUBECONFIGsecret provisioning).