From 51424bea15a0ae424d23f9bcd1146f4e8cdb7244 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Manciot?= Date: Tue, 23 Jun 2026 10:37:20 +0200 Subject: [PATCH 1/2] =?UTF-8?q?Story=2016.5=20=E2=80=94=20helm=20test=20sm?= =?UTF-8?q?oke=20+=20comprehensive=20CI/CD?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the chart-repo CI workflow (.github/workflows/federation-helm.yml) plus the deploy-es.sh / install-sealed-secrets.sh scripts, and the helm test smoke Job. CI redesigned for the chart-only repo: the sbt build-images job is replaced by an image-availability probe that docker-pulls the PUBLIC DockerHub federation + sidecar images and kind-loads them; live-install jobs are gated on image publication and skip-with-warning until published (OQ-1). JFROG creds + setup-java/setup-sbt removed. All static-validation gates (lint, template+kubeconform, golden diff, zero kind:Secret, duckdb-attach discriminator) run unconditionally. Trigger paths updated to softclient4es-federation/**. Smoke = ADBC GetCatalogs (1/3/3 per example). README CI section + Chart.yaml metadata rewritten to match (softclient4es.dev, softclient4es-helm sources, public DockerHub images, private-repo references removed). Closes #9 Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/scripts/deploy-es.sh | 132 +++++ .github/scripts/install-sealed-secrets.sh | 67 +++ .github/workflows/federation-helm.yml | 519 ++++++++++++++++++ softclient4es-federation/README.md | 99 +++- .../templates/deployment.yaml | 13 + .../templates/tests/smoke-test.yaml | 42 +- .../golden/example-heterogeneous-ready.yaml | 42 +- .../tests/golden/example-single-cluster.yaml | 42 +- .../tests/golden/example-three-region.yaml | 42 +- .../tests/golden/ingress-tls.yaml | 42 +- .../tests/golden/secret-auth.yaml | 42 +- .../tests/golden/two-sidecars.yaml | 42 +- softclient4es-federation/values.yaml | 19 +- 13 files changed, 1062 insertions(+), 81 deletions(-) create mode 100755 .github/scripts/deploy-es.sh create mode 100755 .github/scripts/install-sealed-secrets.sh create mode 100644 .github/workflows/federation-helm.yml diff --git a/.github/scripts/deploy-es.sh b/.github/scripts/deploy-es.sh new file mode 100755 index 0000000..2b13c62 --- /dev/null +++ b/.github/scripts/deploy-es.sh @@ -0,0 +1,132 @@ +#!/usr/bin/env bash +# Story 16.5 — deploy single-node Elasticsearch container(s) into a kind cluster, one per +# federation sidecar, reachable at the in-cluster Service name the install job overrides each +# sidecars[].elasticsearch.url to. Used by .github/workflows/federation-helm.yml. +# +# Usage: deploy-es.sh +# topology : single-cluster | three-region | heterogeneous-ready | es-version | secrets +# es-version-list : comma-separated ES major versions, e.g. "8" or "8,8,9" +# +# Service names (must match the --set sidecars[N].elasticsearch.url=... in the install jobs): +# * 1-ES topologies (single-cluster / es-version / secrets) -> Service "es" +# * 3-ES topologies (three-region / heterogeneous-ready) -> "es-us-east-1", "es-eu-west-1", +# "es-ap-south-1" (order = list order) +# +# Each ES runs single-node, security disabled, small heap (fits the 7 GB GitHub runner). +# `vm.max_map_count=262144` MUST already be set on the runner (the workflow does this). +set -euo pipefail + +TOPOLOGY="${1:?usage: deploy-es.sh }" +ES_LIST="${2:?usage: deploy-es.sh }" + +# ES major version -> full image coordinate (CLAUDE.md ES Version Matrix). +es_image() { + case "$1" in + 6) echo "docker.elastic.co/elasticsearch/elasticsearch:6.8.23" ;; + 7) echo "docker.elastic.co/elasticsearch/elasticsearch:7.17.29" ;; + 8) echo "docker.elastic.co/elasticsearch/elasticsearch:8.18.3" ;; + 9) echo "docker.elastic.co/elasticsearch/elasticsearch:9.0.3" ;; + *) echo "unsupported ES version: $1" >&2; exit 1 ;; + esac +} + +# Resolve the Service names for this topology, positionally aligned with the ES list. +case "$TOPOLOGY" in + three-region|heterogeneous-ready) + NAMES=(es-us-east-1 es-eu-west-1 es-ap-south-1) + ;; + single-cluster|es-version|secrets) + NAMES=(es) + ;; + *) + echo "unsupported topology: $TOPOLOGY" >&2; exit 1 ;; +esac + +# Split the comma-separated version list into an array. +IFS=',' read -r -a VERSIONS <<< "$ES_LIST" + +if [ "${#VERSIONS[@]}" -gt "${#NAMES[@]}" ]; then + echo "::error::deploy-es.sh: ${#VERSIONS[@]} ES versions but only ${#NAMES[@]} service name(s) for topology $TOPOLOGY" >&2 + exit 1 +fi + +deploy_one() { + local name="$1" version="$2" image + image="$(es_image "$version")" + echo "Deploying Elasticsearch ${version} as Service '${name}' (${image})" + kubectl apply -f - < +# secret-name : the name of the Secret the controller will materialize (chart references this +# via sidecars[0].elasticsearch.credentialsSecretName), e.g. "es-creds". +# +# Why re-seal in CI (project_sealed_secrets_review_gotchas): +# * The committed examples/sealed-secrets/*.yaml are NON-FUNCTIONAL placeholders — a SealedSecret +# is decryptable ONLY by the specific controller instance that issued the cert. We therefore +# create a fresh plaintext Secret, `kubeseal` it with the live cluster's cert, and apply. +# * The kubeseal CLI version is pinned to the controller chart appVersion (NOT the chart version). +set -euo pipefail + +SECRET_NAME="${1:?usage: install-sealed-secrets.sh }" + +# Pin the controller chart + kubeseal CLI so the CLI/controller versions match (gotcha: +# kubeseal CLI version must equal the controller appVersion, not the Helm chart version). +SEALED_SECRETS_CHART_VERSION="2.16.2" # helm chart version +KUBESEAL_VERSION="0.27.2" # == controller appVersion of chart 2.16.2 + +# 1. Install the controller (repo moved bitnami-labs -> bitnami.github.io). +helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets >/dev/null 2>&1 || \ + helm repo add sealed-secrets https://bitnami.github.io/sealed-secrets +helm repo update sealed-secrets +helm upgrade --install sealed-secrets sealed-secrets/sealed-secrets \ + --namespace kube-system \ + --version "${SEALED_SECRETS_CHART_VERSION}" \ + --set fullnameOverride=sealed-secrets-controller \ + --wait --timeout 180s + +# 2. Install the matching kubeseal CLI. +curl -sSL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz" \ + | tar -xz kubeseal +sudo install -m 0755 kubeseal /usr/local/bin/kubeseal +rm -f kubeseal + +# 3. Wait for the controller to be Ready, then fetch its public cert. +kubectl rollout status deployment/sealed-secrets-controller -n kube-system --timeout=180s +kubeseal --controller-name=sealed-secrets-controller --controller-namespace=kube-system \ + --fetch-cert > /tmp/sealed-secrets-cert.pem + +# 4. Build a plaintext ES-auth Secret (matching the 16.3 es-* key contract), seal it with the +# live cert, and apply the SealedSecret (the plaintext Secret is NEVER applied to the cluster). +kubectl create secret generic "${SECRET_NAME}" \ + --dry-run=client -o yaml \ + --from-literal=es-auth-method=basic \ + --from-literal=es-username=elastic \ + --from-literal=es-password=changeme \ + | kubeseal --cert /tmp/sealed-secrets-cert.pem --format yaml \ + | kubectl apply -f - + +# 5. Wait for the controller to materialize the real Secret from the SealedSecret. +for _ in $(seq 1 30); do + if kubectl get secret "${SECRET_NAME}" >/dev/null 2>&1; then + echo "SealedSecrets materialized Secret '${SECRET_NAME}'." + exit 0 + fi + sleep 2 +done + +echo "::error::SealedSecrets controller did not materialize Secret '${SECRET_NAME}' in time" >&2 +kubectl get sealedsecret,secret 2>/dev/null || true +exit 1 diff --git a/.github/workflows/federation-helm.yml b/.github/workflows/federation-helm.yml new file mode 100644 index 0000000..1c39bff --- /dev/null +++ b/.github/workflows/federation-helm.yml @@ -0,0 +1,519 @@ +# Comprehensive CI for the softclient4es-federation Helm chart (Epic 16, Story 16.5). +# Validates: lint + template + kubeconform + golden-diff (static, hard blocker) AND +# per-topology / per-ES-version / secret-backend / upgrade / uninstall installs on +# ephemeral kind clusters. +# +# This is the dedicated CHART repo (softclient4es-helm) — it is chart-only and CANNOT +# build images via sbt. CI therefore NEVER builds: the live-install jobs `docker pull` +# the PUBLIC DockerHub federation + sidecar images and `kind load` them. checkout@v4, +# `sudo sysctl -w vm.max_map_count=262144` before any Elasticsearch boots. +# +# IMAGE TAGS: the chart's image refs default to the Chart.yaml appVersion. The live jobs +# pull the public DockerHub tags (federation image.tag:"" -> appVersion; sidecar tag -> +# appVersion) and `kind load` them — no per-image --set override, so the federation + +# sidecar tags stay consistent. The committed goldens are appVersion renders, so the +# golden gate renders with NO --set image.tag. +# +# IMAGE AVAILABILITY (OQ-1): neither the federation image nor the sidecar tags are +# published on DockerHub yet. The `image-availability` job probes each with +# `docker manifest inspect`; if ANY required image is absent every live-install job is +# SKIPPED WITH A `::warning::` ANNOTATION (not failed). All static-validation gates run +# UNCONDITIONALLY on every PR with zero external deps. +# +# LICENSE: the multi-sidecar (Pro) tiers are BEST-EFFORT, gated on TWO repo secrets +# together (skip-not-fail if either missing), on top of image availability: (1) secret +# SC4ES_PRO_TEST_JWT, (2) secret SC4ES_TEST_PUBLIC_KEY (injected as +# SOFTCLIENT4ES_LICENSE_PUBLIC_KEY so the verifier resolves the test kid OFFLINE). The +# published federation image must itself be Pro-capable (JWT-verifying SPI on classpath, +# Story 16.1 OQ-5) for the Pro JWT to verify — an OSS-only image falls back to Community. +# single-cluster + per-ES-version + secret backends + uninstall + ALL static gates are +# license-free and run unconditionally (when the images are published). +name: Federation Helm Chart + +on: + workflow_dispatch: + pull_request: + paths: + - 'softclient4es-federation/**' + - '.github/workflows/federation-helm.yml' + - '.github/scripts/deploy-es.sh' + - '.github/scripts/install-sealed-secrets.sh' + push: + branches: ['!main'] + paths: + - 'softclient4es-federation/**' + - '.github/workflows/federation-helm.yml' + - '.github/scripts/deploy-es.sh' + - '.github/scripts/install-sealed-secrets.sh' + # Heavy matrix (per-ES-version + secret backends) can be moved here if PR duration > 30 min + # (see the sharding fallback in the chart README). PRs would then run a reduced matrix. + # schedule: + # - cron: '0 3 * * *' + +permissions: + contents: read + +env: + CHART_DIR: softclient4es-federation + HELM_VERSION: v3.16.3 + KUBECONFORM_VERSION: v0.6.7 + KIND_VERSION: v0.24.0 # kind node images current to K8s 1.31 + KIND_NODE_IMAGE: kindest/node:v1.31.0 + # The chart's image refs default to the Chart.yaml appVersion. The live jobs pull the + # public DockerHub tags at this version so the chart DEFAULTS resolve the kind-loaded images. + IMAGE_TAG: '0.2.0' + FED_IMAGE: softnetwork/softclient4es-federation + SIDECAR_IMAGE_PREFIX: softnetwork/softclient4es # softclient4es-arrow-flight-sql + # A pre-baked ADBC client image avoids the smoke Job's runtime pip install (air-gap). When + # empty the chart smoke Pod falls back to python:3.12-slim + runtime pip (PyPI reachable on + # GitHub-hosted runners). PROPOSED image — see OQ-2 / chart README. + TEST_ADBC_IMAGE: '' + +jobs: + # ── 1. STATIC CHECKS — no cluster, unconditional, hard blocker ───────────── + static-checks: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Set up Helm + uses: azure/setup-helm@v4 + with: + version: ${{ env.HELM_VERSION }} + - name: Install kubeconform + run: | + curl -sSL "https://github.com/yannh/kubeconform/releases/download/${KUBECONFORM_VERSION}/kubeconform-linux-amd64.tar.gz" \ + | sudo tar -xz -C /usr/local/bin kubeconform + - name: helm lint (chart + all examples) + run: | + helm lint "${CHART_DIR}" + for ex in single-cluster three-region heterogeneous-ready; do + f="${CHART_DIR}/examples/${ex}/values.yaml" + if [ -f "$f" ]; then + helm lint "${CHART_DIR}" -f "$f" + else + echo "::warning::missing example $f (Story 16.4 not merged yet — skipping lint -f)" + fi + done + - name: helm template + kubeconform (base + each example) + run: | + render() { helm template fed "${CHART_DIR}" "$@"; } + # base (0-sidecar) always renders + render | kubeconform -strict -summary -kubernetes-version 1.29.0 -ignore-missing-schemas + for ex in single-cluster three-region heterogeneous-ready; do + f="${CHART_DIR}/examples/${ex}/values.yaml" + if [ -f "$f" ]; then + render -f "$f" | kubeconform -strict -summary -kubernetes-version 1.29.0 -ignore-missing-schemas + else + echo "::warning::missing example $f — skipping kubeconform for $ex" + fi + done + - name: golden-file diff (template drift gate) + run: | + fail=0 + # default (16.1) + two-sidecars (16.2) + secret-auth/ingress-tls (16.3) goldens. + # All committed goldens are appVersion renders — render WITH NO --set image.tag. + helm template fed "${CHART_DIR}" \ + | diff -u "${CHART_DIR}/tests/golden/default.yaml" - || { echo "::error::default.yaml golden drift"; fail=1; } + helm template fed "${CHART_DIR}" -f "${CHART_DIR}/tests/values/two-sidecars.yaml" \ + | diff -u "${CHART_DIR}/tests/golden/two-sidecars.yaml" - || { echo "::error::two-sidecars.yaml golden drift"; fail=1; } + helm template fed "${CHART_DIR}" -f "${CHART_DIR}/tests/values/secret-auth.yaml" \ + | diff -u "${CHART_DIR}/tests/golden/secret-auth.yaml" - || { echo "::error::secret-auth.yaml golden drift"; fail=1; } + helm template fed "${CHART_DIR}" -f "${CHART_DIR}/tests/values/ingress-tls.yaml" \ + | diff -u "${CHART_DIR}/tests/golden/ingress-tls.yaml" - || { echo "::error::ingress-tls.yaml golden drift"; fail=1; } + # Per-example goldens (Story 16.4 committed them at tests/golden/example-.yaml). + for ex in single-cluster three-region heterogeneous-ready; do + f="${CHART_DIR}/examples/${ex}/values.yaml" + g="${CHART_DIR}/tests/golden/example-${ex}.yaml" + if [ -f "$f" ] && [ -f "$g" ]; then + helm template fed "${CHART_DIR}" -f "$f" \ + | diff -u "$g" - || { echo "::error::example-${ex}.yaml golden drift"; fail=1; } + else + echo "::warning::missing example or golden for $ex (16.4 not merged) — skipping golden diff" + fi + done + exit $fail + - name: assert chart renders ZERO kind:Secret (16.3 contract) + run: | + n=$(helm template fed "${CHART_DIR}" | grep -c '^kind: Secret' || true) + [ "$n" = "0" ] || { echo "::error::chart must NOT render a kind:Secret (got $n)"; exit 1; } + - name: heterogeneous-ready discriminator (16.4 A2b — only signal vs three-region) + run: | + # three-region and heterogeneous-ready goldens are BYTE-IDENTICAL by design (Helm + # strips the R2b duckdb-attach comment block). The ONLY signal distinguishing them is + # the commented duckdb-attach preview in the SOURCE values.yaml. Guard against a + # regression that copies one example over the other (passes golden + smoke undetected). + het="${CHART_DIR}/examples/heterogeneous-ready/values.yaml" + tr="${CHART_DIR}/examples/three-region/values.yaml" + if [ -f "$het" ] && [ -f "$tr" ]; then + nh=$(grep -c 'type = "duckdb-attach"' "$het" || true) + nt=$(grep -c 'type = "duckdb-attach"' "$tr" || true) + [ "$nh" = "3" ] || { echo "::error::heterogeneous-ready must keep 3 commented duckdb-attach previews (got $nh)"; exit 1; } + [ "$nt" = "0" ] || { echo "::error::three-region must have 0 duckdb-attach entries (got $nt)"; exit 1; } + else + echo "::warning::examples missing (16.4 not merged) — skipping A2b discriminator" + fi + + # ── 2. IMAGE AVAILABILITY — probe the PUBLIC DockerHub images, prerequisite for installs ─ + # This chart repo cannot build images (no sbt). The federation image + the four sidecar tags + # are pulled from public DockerHub. They are NOT yet published (OQ-1), so probe each tag with + # `docker manifest inspect`; if ANY is absent, the live-install jobs SKIP WITH A WARNING. + image-availability: + runs-on: ubuntu-latest + outputs: + images_published: ${{ steps.probe.outputs.images_published }} # gate for ALL live-install jobs + steps: + - name: Probe public DockerHub images (federation + 4 sidecars) + id: probe + run: | + published=true + probe() { + if docker manifest inspect "$1" >/dev/null 2>&1; then + echo "found: $1" + else + echo "::warning::image not published on DockerHub: $1 — live-install jobs will be SKIPPED (OQ-1)" + published=false + fi + } + probe "docker.io/${FED_IMAGE}:${IMAGE_TAG}" + for v in 6 7 8 9; do + probe "docker.io/${SIDECAR_IMAGE_PREFIX}${v}-arrow-flight-sql:${IMAGE_TAG}" + done + if [ "$published" != "true" ]; then + echo "::warning::one or more federation/sidecar images are not yet published (OQ-1) — skipping all live kind-install jobs; static gates still ran" + fi + echo "images_published=$published" >> "$GITHUB_OUTPUT" + + # ── 3. PER-TOPOLOGY INSTALL — single-cluster (license-free) + 3-cluster (Pro) ─ + install-topology: + runs-on: ubuntu-latest + needs: [static-checks, image-availability] + if: ${{ needs.image-availability.outputs.images_published == 'true' }} + strategy: + fail-fast: false + matrix: + include: + - example: single-cluster + es: '8' # one ES8, hostname `es` + expect_catalogs: 1 + license: false # 1 sidecar -> Community, no license (FACT C) + - example: three-region + es: '8,8,9' # two ES8 + one ES9 (mixed-version) + expect_catalogs: 3 + license: true # 3 sidecars -> Pro JWT required (FACT C/F) + - example: heterogeneous-ready + es: '8,8,9' # 3 ES (R2b placeholders inactive in R1) + expect_catalogs: 3 + license: true + steps: + - uses: actions/checkout@v4 + - name: Set vm.max_map_count (Elasticsearch requirement) + run: sudo sysctl -w vm.max_map_count=262144 + - name: Skip if example missing (Story 16.4 not merged) + id: guard + run: | + f="${CHART_DIR}/examples/${{ matrix.example }}/values.yaml" + if [ ! -f "$f" ]; then + echo "::warning::$f missing — Story 16.4 not merged; skipping install" + echo "skip=true" >> "$GITHUB_OUTPUT" + fi + - name: Skip multi-sidecar install when no Pro test license (FACT C + FACT F, best-effort tier) + id: lic + if: ${{ matrix.license && steps.guard.outputs.skip != 'true' }} + # The 3-sidecar tier needs BOTH: a Pro test JWT and its public verification key (so the + # verifier resolves the test kid OFFLINE). Missing EITHER → SKIP (annotation), never fail. + # (The published federation image must itself be Pro-capable — 16.1 OQ-5 — for the JWT to + # verify; an OSS-only image silently falls back to Community and CrashLoops at 3 clusters.) + run: | + if [ -z "${{ secrets.SC4ES_PRO_TEST_JWT }}" ] || [ -z "${{ secrets.SC4ES_TEST_PUBLIC_KEY }}" ]; then + echo "::warning::missing SC4ES_PRO_TEST_JWT and/or SC4ES_TEST_PUBLIC_KEY — ${{ matrix.example }} (3 sidecars) needs a Pro JWT + its public key (FACT F #2); skipping (best-effort)" + echo "skip=true" >> "$GITHUB_OUTPUT" + fi + - name: Set up Helm + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + uses: azure/setup-helm@v4 + with: { version: "${{ env.HELM_VERSION }}" } + - name: Create kind cluster + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + uses: helm/kind-action@v1.10.0 + with: + version: ${{ env.KIND_VERSION }} + node_image: ${{ env.KIND_NODE_IMAGE }} + cluster_name: fed-${{ matrix.example }} + - name: Pull public images + kind load (federation + sidecars) + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + run: | + docker pull "docker.io/${FED_IMAGE}:${IMAGE_TAG}" + kind load docker-image "docker.io/${FED_IMAGE}:${IMAGE_TAG}" --name "fed-${{ matrix.example }}" + for v in $(echo '${{ matrix.es }}' | tr ',' '\n' | sort -u); do + img="docker.io/${SIDECAR_IMAGE_PREFIX}${v}-arrow-flight-sql:${IMAGE_TAG}" + docker pull "$img" + kind load docker-image "$img" --name "fed-${{ matrix.example }}" + done + - name: Deploy ES container(s) per sidecar (FACT D — reachable ES required) + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + run: ./.github/scripts/deploy-es.sh "${{ matrix.example }}" '${{ matrix.es }}' + - name: Create Pro license + public-key Secret (multi-sidecar only — FACT F #2) + if: ${{ matrix.license && steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + run: | + # ONE Secret carries both the JWT (license-key) and the public verification JWK + # (license-public-key → SOFTCLIENT4ES_LICENSE_PUBLIC_KEY, the air-gap path in + # LicenseKeyVerifier.loadPublicKey Step 3 — without it the test kid never resolves). + kubectl create secret generic sc4es-license \ + --from-literal=license-key="${{ secrets.SC4ES_PRO_TEST_JWT }}" \ + --from-literal=license-public-key="${{ secrets.SC4ES_TEST_PUBLIC_KEY }}" + # The three-region example pins per-region license.secretName=sc4es-pro-license; also + # create that name so its bearer-auth sidecars + the federation share the test token. + kubectl create secret generic sc4es-pro-license \ + --from-literal=license-key="${{ secrets.SC4ES_PRO_TEST_JWT }}" \ + --from-literal=license-public-key="${{ secrets.SC4ES_TEST_PUBLIC_KEY }}" || true + - name: helm install + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + run: | + # The example values point sidecars[].elasticsearch.url at external https://*.example.com + # placeholders. deploy-es.sh stands up in-cluster ES Services; override each URL to the + # matching plaintext in-cluster Service. license/public-key are pointed at sc4es-license + # for the multi-sidecar tier. + extra="" + if [ "${{ matrix.license }}" = "true" ]; then + extra="--set license.secretName=sc4es-license --set license.publicKeySecretName=sc4es-license" + fi + # The three-region/heterogeneous examples set sidecars[].auth.method=bearer pointing at + # per-region sc4es-arrow-* Secrets that this CI does NOT create (the dedicated + # install-secrets job covers Secret-backed auth). Override the federation↔sidecar hop to + # auth.method=none so catalog discovery is what's under test here, not bearer auth — else + # the sidecars enforce bearer with an absent (optional) token and discovery fails. + urls="" + case "${{ matrix.example }}" in + single-cluster) + urls="--set sidecars[0].elasticsearch.url=http://es:9200" ;; + three-region|heterogeneous-ready) + urls="--set sidecars[0].elasticsearch.url=http://es-us-east-1:9200 \ + --set sidecars[1].elasticsearch.url=http://es-eu-west-1:9200 \ + --set sidecars[2].elasticsearch.url=http://es-ap-south-1:9200 \ + --set sidecars[0].auth.method=none \ + --set sidecars[1].auth.method=none \ + --set sidecars[2].auth.method=none" ;; + esac + helm install fed "${CHART_DIR}" \ + -f "${CHART_DIR}/examples/${{ matrix.example }}/values.yaml" \ + ${TEST_ADBC_IMAGE:+--set test.image=$TEST_ADBC_IMAGE} \ + $urls $extra + - name: Wait all pods Ready (sidecars + federation — FACT D) + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + run: kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=softclient4es --timeout=300s + - name: helm test (GetCatalogs == expected — FACT A) + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + run: helm test fed --timeout 180s + - name: Diagnostics on failure + if: ${{ failure() }} + run: | + kubectl get pods -o wide || true + kubectl logs -l app.kubernetes.io/component=federation --tail=200 || true + kubectl logs -l app.kubernetes.io/component=sidecar --tail=100 || true + kubectl logs -l app.kubernetes.io/name=softclient4es-federation -c show-catalogs --tail=100 || true + + # ── 4. PER-ES-VERSION SIDECAR MATRIX — 1 sidecar each (license-free) ──────── + install-es-version: + runs-on: ubuntu-latest + needs: [static-checks, image-availability] + if: ${{ needs.image-availability.outputs.images_published == 'true' }} + strategy: + fail-fast: false + matrix: + es: [6, 7, 8, 9] + steps: + - uses: actions/checkout@v4 + - name: Set vm.max_map_count (Elasticsearch requirement) + run: sudo sysctl -w vm.max_map_count=262144 + - name: Set up Helm + uses: azure/setup-helm@v4 + with: { version: "${{ env.HELM_VERSION }}" } + - name: Create kind cluster + uses: helm/kind-action@v1.10.0 + with: { version: "${{ env.KIND_VERSION }}", node_image: "${{ env.KIND_NODE_IMAGE }}", cluster_name: "fed-es${{ matrix.es }}" } + - name: Pull public images + kind load (federation + ES${{ matrix.es }} sidecar) + run: | + docker pull "docker.io/${FED_IMAGE}:${IMAGE_TAG}" + kind load docker-image "docker.io/${FED_IMAGE}:${IMAGE_TAG}" --name "fed-es${{ matrix.es }}" + img="docker.io/${SIDECAR_IMAGE_PREFIX}${{ matrix.es }}-arrow-flight-sql:${IMAGE_TAG}" + docker pull "$img" + kind load docker-image "$img" --name "fed-es${{ matrix.es }}" + - name: Deploy ES ${{ matrix.es }} + run: ./.github/scripts/deploy-es.sh es-version '${{ matrix.es }}' + - name: helm install (1 sidecar, ES ${{ matrix.es }}, license-free) + run: | + helm install fed "${CHART_DIR}" \ + ${TEST_ADBC_IMAGE:+--set test.image=$TEST_ADBC_IMAGE} \ + --set 'sidecars[0].name=primary' \ + --set "sidecars[0].elasticsearchVersion=${{ matrix.es }}" \ + --set 'sidecars[0].elasticsearch.url=http://es:9200' \ + --set 'sidecars[0].default=true' + - name: Wait Ready + helm test (== 1) + run: | + kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=softclient4es --timeout=300s + helm test fed --timeout 180s + - name: Diagnostics on failure + if: ${{ failure() }} + run: | + kubectl get pods -o wide || true + kubectl logs -l app.kubernetes.io/component=federation --tail=200 || true + kubectl logs -l app.kubernetes.io/component=sidecar --tail=100 || true + + # ── 5. SECRET MANAGEMENT — raw Secret + SealedSecrets (1 sidecar, license-free) ─ + install-secrets: + runs-on: ubuntu-latest + needs: [static-checks, image-availability] + if: ${{ needs.image-availability.outputs.images_published == 'true' }} + strategy: + fail-fast: false + matrix: + backend: [raw-secret, sealed-secrets] + steps: + - uses: actions/checkout@v4 + - name: Set vm.max_map_count (Elasticsearch requirement) + run: sudo sysctl -w vm.max_map_count=262144 + - name: Set up Helm + uses: azure/setup-helm@v4 + with: { version: "${{ env.HELM_VERSION }}" } + - name: Create kind cluster + uses: helm/kind-action@v1.10.0 + with: { version: "${{ env.KIND_VERSION }}", node_image: "${{ env.KIND_NODE_IMAGE }}", cluster_name: "fed-${{ matrix.backend }}" } + - name: Pull public images + kind load (federation + ES8 sidecar) + run: | + docker pull "docker.io/${FED_IMAGE}:${IMAGE_TAG}" + kind load docker-image "docker.io/${FED_IMAGE}:${IMAGE_TAG}" --name "fed-${{ matrix.backend }}" + img="docker.io/${SIDECAR_IMAGE_PREFIX}8-arrow-flight-sql:${IMAGE_TAG}" + docker pull "$img" + kind load docker-image "$img" --name "fed-${{ matrix.backend }}" + - name: Deploy ES 8 + run: ./.github/scripts/deploy-es.sh secrets '8' + - name: Install SealedSecrets controller + materialize the ES Secret (sealed-secrets only) + if: ${{ matrix.backend == 'sealed-secrets' }} + run: ./.github/scripts/install-sealed-secrets.sh es-creds + - name: Create the ES auth Secret (raw-secret only) + if: ${{ matrix.backend == 'raw-secret' }} + run: | + kubectl create secret generic es-creds \ + --from-literal=es-auth-method=basic \ + --from-literal=es-username=elastic --from-literal=es-password=changeme + - name: helm install (1 sidecar, Secret-backed ES auth) + run: | + helm install fed "${CHART_DIR}" \ + ${TEST_ADBC_IMAGE:+--set test.image=$TEST_ADBC_IMAGE} \ + --set 'sidecars[0].name=primary' --set 'sidecars[0].elasticsearchVersion=8' \ + --set 'sidecars[0].elasticsearch.url=http://es:9200' \ + --set 'sidecars[0].elasticsearch.credentialsSecretName=es-creds' \ + --set 'sidecars[0].default=true' + - name: Wait Ready + assert ES env injected + helm test + run: | + kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=softclient4es --timeout=300s + # Assert the Secret-backed env actually reached the sidecar container (16.3 AC9). + kubectl get pod -l app.kubernetes.io/component=sidecar \ + -o jsonpath='{.items[0].spec.containers[0].env[*].name}' | grep -q ELASTIC_CREDENTIALS_USERNAME + helm test fed --timeout 180s + - name: Diagnostics on failure + if: ${{ failure() }} + run: | + kubectl get pods -o wide || true + kubectl logs -l app.kubernetes.io/component=federation --tail=200 || true + kubectl logs -l app.kubernetes.io/component=sidecar --tail=100 || true + + # ── 6. UPGRADE — single-cluster -> three-region (Pro license, best-effort) ── + upgrade: + runs-on: ubuntu-latest + needs: [static-checks, image-availability] + if: ${{ needs.image-availability.outputs.images_published == 'true' }} + steps: + - uses: actions/checkout@v4 + - name: Set vm.max_map_count (Elasticsearch requirement) + run: sudo sysctl -w vm.max_map_count=262144 + - name: Skip if examples missing or no Pro test license (FACT C + FACT F) + id: guard + # Upgrade ends at 3 clusters → same Pro gate as install-topology: JWT secret AND + # public-key secret (+ the published image must be Pro-capable, 16.1 OQ-5). + run: | + if [ ! -f "${CHART_DIR}/examples/single-cluster/values.yaml" ] || [ ! -f "${CHART_DIR}/examples/three-region/values.yaml" ] \ + || [ -z "${{ secrets.SC4ES_PRO_TEST_JWT }}" ] || [ -z "${{ secrets.SC4ES_TEST_PUBLIC_KEY }}" ]; then + echo "::warning::upgrade test needs both examples (16.4) + SC4ES_PRO_TEST_JWT + SC4ES_TEST_PUBLIC_KEY (and a Pro-capable image, FACT F); skipping" + echo "skip=true" >> "$GITHUB_OUTPUT" + fi + - name: Set up Helm + if: ${{ steps.guard.outputs.skip != 'true' }} + uses: azure/setup-helm@v4 + with: { version: "${{ env.HELM_VERSION }}" } + - name: Create kind cluster + if: ${{ steps.guard.outputs.skip != 'true' }} + uses: helm/kind-action@v1.10.0 + with: { version: "${{ env.KIND_VERSION }}", node_image: "${{ env.KIND_NODE_IMAGE }}", cluster_name: fed-upgrade } + - name: Pull public images + kind load + ES + license + install single-cluster + if: ${{ steps.guard.outputs.skip != 'true' }} + run: | + docker pull "docker.io/${FED_IMAGE}:${IMAGE_TAG}" + kind load docker-image "docker.io/${FED_IMAGE}:${IMAGE_TAG}" --name fed-upgrade + for v in 8 9; do + img="docker.io/${SIDECAR_IMAGE_PREFIX}${v}-arrow-flight-sql:${IMAGE_TAG}" + docker pull "$img" + kind load docker-image "$img" --name fed-upgrade + done + ./.github/scripts/deploy-es.sh three-region '8,8,9' + # single-cluster needs a `es` Service too (its sidecar URL is overridden to http://es:9200). + ./.github/scripts/deploy-es.sh secrets '8' + for n in sc4es-license sc4es-pro-license; do + kubectl create secret generic "$n" \ + --from-literal=license-key="${{ secrets.SC4ES_PRO_TEST_JWT }}" \ + --from-literal=license-public-key="${{ secrets.SC4ES_TEST_PUBLIC_KEY }}" || true + done + helm install fed "${CHART_DIR}" -f "${CHART_DIR}/examples/single-cluster/values.yaml" \ + ${TEST_ADBC_IMAGE:+--set test.image=$TEST_ADBC_IMAGE} \ + --set sidecars[0].elasticsearch.url=http://es:9200 + kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=softclient4es --timeout=300s + - name: helm upgrade -> three-region + assert topology change + if: ${{ steps.guard.outputs.skip != 'true' }} + run: | + before=$(kubectl get deploy -l app.kubernetes.io/component=sidecar --no-headers | wc -l) + # Same auth.method=none override as install-topology: this CI tests the topology change + # + catalog count, not the per-region bearer Secrets (which it does not create). + helm upgrade fed "${CHART_DIR}" -f "${CHART_DIR}/examples/three-region/values.yaml" \ + ${TEST_ADBC_IMAGE:+--set test.image=$TEST_ADBC_IMAGE} \ + --set license.secretName=sc4es-license --set license.publicKeySecretName=sc4es-license \ + --set sidecars[0].elasticsearch.url=http://es-us-east-1:9200 \ + --set sidecars[1].elasticsearch.url=http://es-eu-west-1:9200 \ + --set sidecars[2].elasticsearch.url=http://es-ap-south-1:9200 \ + --set sidecars[0].auth.method=none \ + --set sidecars[1].auth.method=none \ + --set sidecars[2].auth.method=none + kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=softclient4es --timeout=300s + after=$(kubectl get deploy -l app.kubernetes.io/component=sidecar --no-headers | wc -l) + [ "$after" -gt "$before" ] || { echo "::error::upgrade did not add sidecar Deployments ($before -> $after)"; exit 1; } + helm test fed --timeout 180s + - name: Diagnostics on failure + if: ${{ failure() && steps.guard.outputs.skip != 'true' }} + run: | + kubectl get pods,deploy -o wide || true + kubectl logs -l app.kubernetes.io/component=federation --tail=200 || true + + # ── 7. UNINSTALL — clean teardown, no leftovers (license-free, 0 sidecars) ── + uninstall: + runs-on: ubuntu-latest + needs: [static-checks, image-availability] + if: ${{ needs.image-availability.outputs.images_published == 'true' }} + steps: + - uses: actions/checkout@v4 + - name: Set up Helm + uses: azure/setup-helm@v4 + with: { version: "${{ env.HELM_VERSION }}" } + - name: Create kind cluster + uses: helm/kind-action@v1.10.0 + with: { version: "${{ env.KIND_VERSION }}", node_image: "${{ env.KIND_NODE_IMAGE }}", cluster_name: fed-uninstall } + - name: Pull public federation image + kind load + run: | + docker pull "docker.io/${FED_IMAGE}:${IMAGE_TAG}" + kind load docker-image "docker.io/${FED_IMAGE}:${IMAGE_TAG}" --name fed-uninstall + - name: Install (0-sidecar — license-free, no ES needed) then uninstall + run: | + helm install fed "${CHART_DIR}" + kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=softclient4es-federation --timeout=180s + helm uninstall fed + sleep 5 + leftover=$(kubectl get all,configmap -l app.kubernetes.io/instance=fed --no-headers 2>/dev/null | grep -v '^kubernetes ' | wc -l) + [ "$leftover" = "0" ] || { echo "::error::helm uninstall left $leftover resources"; kubectl get all,configmap -l app.kubernetes.io/instance=fed; exit 1; } diff --git a/softclient4es-federation/README.md b/softclient4es-federation/README.md index 6c7c234..78a8427 100644 --- a/softclient4es-federation/README.md +++ b/softclient4es-federation/README.md @@ -88,6 +88,8 @@ helm uninstall fed | `federation.upgradeUrl` | `https://portal.softclient4es.com/pricing` | `FEDERATION_UPGRADE_URL`. | | `telemetry.enabled` | `true` | `SOFTCLIENT4ES_TELEMETRY_ENABLED` daily-ping opt-out (`false` opts out). | | `license.secretName` | `""` | Secret holding license/API key; empty = Community. | +| `license.publicKeySecretName` | `""` | Secret holding the Ed25519 public JWK for OFFLINE license verification → `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY`; empty = use JWKS fetch. | +| `license.publicKeyKey` | `license-public-key` | Data key within `license.publicKeySecretName`. | | `service.type` | `ClusterIP` | Service type. | | `service.port` | `32020` | `FEDERATION_PORT` (Flight SQL); the only port exposed by the Service. | | `resources` | req `1Gi`/`500m`, lim `2Gi`/`1000m` | Container resource requests/limits. | @@ -104,6 +106,7 @@ helm uninstall fed | `sidecars` | `[]` | Per-ES-version sidecars — see the "Sidecars" section below. | | `sidecarDefaults` | (see `values.yaml`) | Shared resource/probe/security defaults applied to every sidecar. | | `test.image` | `""` (→ `python:3.12-slim`) | Image for the `helm test` smoke Job; pin a pre-baked ADBC image for air-gapped clusters. | +| `test.adbcVersion` | `1.6.0` | ADBC driver version the smoke Job `pip install`s at runtime (when `test.image` is not pre-baked). | ## Sidecars — federating one or more Elasticsearch clusters @@ -182,7 +185,21 @@ exec is the compat path — see the operator guide). With sidecars configured, `helm test fed` runs a Job that connects to the federation Flight SQL endpoint and asserts `GetCatalogs` returns one catalog per sidecar (`len(sidecars)`). The 2-sidecar smoke requires a Pro/Enterprise license (the quota -gate above) and reachable backing ES for each sidecar. +gate above) and reachable backing ES for each sidecar. The Job **retries the connect + +`GetCatalogs` for up to ~60 s** — the federation is NotReady until it has discovered every +downstream (gRPC readiness is all-or-nothing), so a fresh install needs a few seconds. + +The test Pod defaults to `python:3.12-slim` and `pip install`s the ADBC driver at runtime +(needs PyPI reachability). For air-gapped or rate-limited clusters, pin a pre-baked ADBC image +with `--set test.image=` (and `--set test.adbcVersion=` to control the driver version +when the runtime install IS used). This is the same Job CI runs. + +> **Offline license verification (`license.publicKeySecretName`).** When the federation must +> verify a license JWT WITHOUT reaching the license server's JWKS endpoint (air-gapped clusters, +> or a JWT whose `kid` is not in the prod JWKS), set `license.publicKeySecretName` to a Secret +> whose `license.publicKeyKey` data key holds the matching Ed25519 public JWK. It is mounted as +> `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY` (the air-gap path in the license verifier). Leave empty (the +> default) to use the normal JWKS fetch — it renders nothing, so the golden render is unaffected. ## Secrets, TLS & Ingress @@ -317,3 +334,83 @@ the install command, and the `SHOW CATALOGS` smoke expectation: Install any of them with `helm install softclient4es-federation softclient4es-federation -f examples//values.yaml` (create the referenced Secrets first — see each example's README). + +## CI / CD coverage (Story 16.5) + +Every PR touching `softclient4es-federation/**` runs +[`.github/workflows/federation-helm.yml`](../.github/workflows/federation-helm.yml): static +checks (lint + template + `kubeconform -strict` + golden-file diff), an image-availability +prerequisite (pulls the public DockerHub federation + sidecar images), and per-topology / +per-ES-version / secret-backend / upgrade / uninstall installs on ephemeral `kind` clusters. +The smoke test is the chart's own `helm test` Job (ADBC Flight SQL `GetCatalogs`, asserting one +catalog per sidecar) — customers run the exact same `helm test fed` post-install that CI runs. + +This chart repo is **chart-only** (no sbt build): CI never builds images. The live-install jobs +`docker pull` the public DockerHub images and `kind load` them. Until those images are published +(OQ-1), the live-install jobs **probe image availability and skip with a CI annotation** rather +than fail — the static-validation gates run unconditionally on every PR with zero external deps. + +### Tested vs best-effort matrix + +| Scenario | Gate | Tier | License | Notes | +|---|---|---|---|---| +| `helm lint` (chart + each example) | static | **tested** (blocker) | none | unconditional | +| `helm template` + `kubeconform -strict` | static | **tested** (blocker) | none | per example | +| Golden-file diff (template drift) | static | **tested** (blocker) | none | per example + the 16.1/16.2/16.3 goldens | +| Zero `kind: Secret` rendered | static | **tested** (blocker) | none | 16.3 contract | +| `heterogeneous-ready` discriminator | static | **tested** (blocker) | none | greps the `type = "duckdb-attach"` marker (the only signal vs `three-region` — the goldens are byte-identical) | +| Install single-cluster (1×ES8) | kind | **tested** | none (Community) | `GetCatalogs` == 1 | +| Install three-region (2×ES8 + 1×ES9) | kind | **best-effort\*** | **Pro** | mixed-version; `== 3` | +| Install heterogeneous-ready (3×ES) | kind | **best-effort\*** | **Pro** | R2b placeholders inactive; `== 3` | +| Per-ES-version sidecar (6/7/8/9) | kind | **tested** | none | 1 sidecar each; `== 1` | +| Secret backend: raw K8s Secret | kind | **tested** | none | 1 sidecar Secret-backed | +| Secret backend: SealedSecrets | kind | **tested** | none | controller installed + re-sealed in-cluster | +| Secret backend: External Secrets Operator (ESO) | — | **best-effort / documented** | — | needs an external store; examples shipped (16.3), not CI-run | +| Secret backend: Vault Agent Injector | — | **best-effort / documented** | — | needs Vault; documented (16.3) | +| Upgrade single → three-region | kind | **best-effort\*** | **Pro** | topology change asserted (+2 sidecar Deployments) | +| Uninstall clean | kind | **tested** | none | `kubectl get all` empty | + +\* The multi-cluster (Pro) tiers run only when ALL of: (1) the federation image bundles the +JWT-verifying SPI (a **Pro-capable** image — an OSS-only image ships only the Community SPI and +cannot verify ANY Pro JWT), (2) the `SC4ES_PRO_TEST_JWT` repo secret, and (3) the +`SC4ES_TEST_PUBLIC_KEY` repo secret (injected as `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY` via +`license.publicKeySecretName` so the JWT verifies offline) are present. Otherwise they are +**skipped with a CI annotation** (not a failure). A **single-cluster** federation is license-FREE +(Community `maxClusters=1`) and is always tested; the static three-region/heterogeneous golden +proves the mixed-version RENDER on every PR even when the live multi-cluster install is skipped. + +### Image tags in CI + +The live-install jobs resolve the chart's DEFAULT image refs (federation `image.tag:""` → +`appVersion`; sidecar tag → `appVersion`), `docker pull` those public DockerHub tags, and +`kind load` them — no per-image `--set image.tag` override, so the federation + sidecar tags +stay consistent with the committed goldens (which are `appVersion` renders). Until the images +are published (OQ-1), an availability probe (`docker manifest inspect`) gates each live-install +job: if a required image is absent the job is **skipped with a `::warning::` annotation**, never +failed. The golden gate renders with NO `--set image.tag`. + +### CI failure modes (troubleshooting) + +| CI job fails with… | Likely cause | Fix | +|---|---|---| +| golden-file diff non-empty | a template change was not regenerated | run the regen commands above; commit the new golden if the change is intentional | +| `kubeconform` invalid resource | a manifest field renamed/typo (e.g. `replicaCount` vs `replicas`) | fix the template; `replicas` is the Deployment field | +| `heterogeneous-ready` discriminator fails | `three-region` and `heterogeneous-ready` overlays drifted (one copied over the other) | restore the commented R2b `duckdb-attach` preview in `heterogeneous-ready` (the goldens are byte-identical — this grep is the only signal) | +| federation pod CrashLoopBackOff (3 sidecars, license supplied but Community at runtime) | the image lacks the JWT SPI — it cannot verify the Pro JWT → falls back to Community → `maxClusters=1` exceeded | use a Pro-CAPABLE federation image (JWT SPI on classpath, 16.1 OQ-5); injecting a JWT into an OSS-only image does nothing | +| federation pod CrashLoopBackOff (`InvalidLicense: Unknown key ID: …`) | the JWT verification key didn't resolve (no JWKS entry for the kid AND no `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY`) | set `license.publicKeySecretName` → `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY` (the offline verifier path), or ensure the license-server JWKS carries the kid | +| federation pod CrashLoopBackOff (3 sidecars, no license at all) | Community `maxClusters=1` exceeded | supply a Pro/Enterprise license (`license.secretName`) — by design | +| federation pod CrashLoopBackOff (`validate()` / `FlightCredentials`) | Secret-backed cred didn't arrive (wrong key / ESO sync lag) | check the Secret exists + keys match the contract table above; self-heals on next restart | +| federation NotReady, smoke connect-refused | a sidecar's backing ES is down/unreachable (all-or-nothing gRPC readiness) | ensure every sidecar's ES is reachable; or set `federation.probes.useGrpc=false` for partial availability | +| `helm test` count mismatch | a sidecar failed discovery (ES down, or wrong `elasticsearch.url`) | check sidecar + ES logs; verify each sidecar's `elasticsearch.url` resolves in-cluster | +| image pull error (federation OR sidecar) | the public DockerHub tag is not published yet (OQ-1) | the availability probe should skip the live-install job until the image is published; once published, the job `docker pull`s + `kind load`s the public tag | +| live-install job skipped with a `::warning::` | the required public DockerHub image is not yet published (OQ-1) | expected until release; the static-validation gates still run unconditionally | +| ES pod crash (`max virtual memory areas …`) | `vm.max_map_count` too low | `sudo sysctl -w vm.max_map_count=262144` on the runner before ES starts (the workflow does this) | + +### Duration & sharding + +Static checks < 2 min; image pull ~1–2 min; each install job ~6–9 min (when the public images +exist — otherwise the live jobs skip in seconds). With the matrix in parallel, wall-clock ≈ +image-pull + slowest install ≈ ~12 min (< 30 min budget). If runner contention serializes the +matrix past 30 min, move the per-ES-version + secret-backend jobs to a `schedule:` nightly +trigger (uncomment the `schedule:` block in the workflow) and keep PRs to static-checks + +single-cluster + three-region + uninstall. diff --git a/softclient4es-federation/templates/deployment.yaml b/softclient4es-federation/templates/deployment.yaml index 28005e0..a8dfc33 100644 --- a/softclient4es-federation/templates/deployment.yaml +++ b/softclient4es-federation/templates/deployment.yaml @@ -78,6 +78,19 @@ spec: key: {{ .Values.license.apiKeyKey | default "api-key" }} optional: true {{- end }} + {{- /* Story 16.5 (FACT F #2): offline license public-key for air-gapped / + test-signed JWT verification (LicenseKeyVerifier.loadPublicKey Step 3). + Gated INDEPENDENTLY of license.secretName so the multi-sidecar CI can + point it at the same Secret. Renders nothing at the "" default, so the + golden and every example are byte-stable. */}} + {{- if .Values.license.publicKeySecretName }} + - name: SOFTCLIENT4ES_LICENSE_PUBLIC_KEY + valueFrom: + secretKeyRef: + name: {{ .Values.license.publicKeySecretName }} + key: {{ .Values.license.publicKeyKey | default "license-public-key" }} + optional: true + {{- end }} {{- if gt (len .Values.sidecars) 0 }} # Point Typesafe Config at the mounted ConfigMap so ConfigFactory.load() # picks up the `servers` map (which has NO env override). The native-packager diff --git a/softclient4es-federation/templates/tests/smoke-test.yaml b/softclient4es-federation/templates/tests/smoke-test.yaml index 239955c..2157bd9 100644 --- a/softclient4es-federation/templates/tests/smoke-test.yaml +++ b/softclient4es-federation/templates/tests/smoke-test.yaml @@ -12,24 +12,44 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: {{ .Values.test.image | default "python:3.12-slim" | quote }} command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql=={{ .Values.test.adbcVersion | default "1.6.0" }} \ + adbc_driver_manager=={{ .Values.test.adbcVersion | default "1.6.0" }} python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "{{ include "softclient4es-federation.fullname" . }}" - conn = dbapi.connect(f"grpc://{host}:{{ .Values.service.port }}") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:{{ .Values.service.port }}" expected = {{ len .Values.sidecars }} - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY {{- end }} diff --git a/softclient4es-federation/tests/golden/example-heterogeneous-ready.yaml b/softclient4es-federation/tests/golden/example-heterogeneous-ready.yaml index e305683..32e34bf 100644 --- a/softclient4es-federation/tests/golden/example-heterogeneous-ready.yaml +++ b/softclient4es-federation/tests/golden/example-heterogeneous-ready.yaml @@ -926,23 +926,43 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: "python:3.12-slim" command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql==1.6.0 \ + adbc_driver_manager==1.6.0 python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "fed-softclient4es-federation" - conn = dbapi.connect(f"grpc://{host}:32020") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:32020" expected = 3 - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY diff --git a/softclient4es-federation/tests/golden/example-single-cluster.yaml b/softclient4es-federation/tests/golden/example-single-cluster.yaml index dbafbab..a94e4b8 100644 --- a/softclient4es-federation/tests/golden/example-single-cluster.yaml +++ b/softclient4es-federation/tests/golden/example-single-cluster.yaml @@ -399,23 +399,43 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: "python:3.12-slim" command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql==1.6.0 \ + adbc_driver_manager==1.6.0 python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "fed-softclient4es-federation" - conn = dbapi.connect(f"grpc://{host}:32020") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:32020" expected = 1 - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY diff --git a/softclient4es-federation/tests/golden/example-three-region.yaml b/softclient4es-federation/tests/golden/example-three-region.yaml index e305683..32e34bf 100644 --- a/softclient4es-federation/tests/golden/example-three-region.yaml +++ b/softclient4es-federation/tests/golden/example-three-region.yaml @@ -926,23 +926,43 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: "python:3.12-slim" command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql==1.6.0 \ + adbc_driver_manager==1.6.0 python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "fed-softclient4es-federation" - conn = dbapi.connect(f"grpc://{host}:32020") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:32020" expected = 3 - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY diff --git a/softclient4es-federation/tests/golden/ingress-tls.yaml b/softclient4es-federation/tests/golden/ingress-tls.yaml index d86c77d..c124858 100644 --- a/softclient4es-federation/tests/golden/ingress-tls.yaml +++ b/softclient4es-federation/tests/golden/ingress-tls.yaml @@ -406,23 +406,43 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: "python:3.12-slim" command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql==1.6.0 \ + adbc_driver_manager==1.6.0 python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "fed-softclient4es-federation" - conn = dbapi.connect(f"grpc://{host}:32020") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:32020" expected = 1 - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY diff --git a/softclient4es-federation/tests/golden/secret-auth.yaml b/softclient4es-federation/tests/golden/secret-auth.yaml index 20ca27a..bc47a30 100644 --- a/softclient4es-federation/tests/golden/secret-auth.yaml +++ b/softclient4es-federation/tests/golden/secret-auth.yaml @@ -438,23 +438,43 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: "python:3.12-slim" command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql==1.6.0 \ + adbc_driver_manager==1.6.0 python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "fed-softclient4es-federation" - conn = dbapi.connect(f"grpc://{host}:32020") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:32020" expected = 1 - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY diff --git a/softclient4es-federation/tests/golden/two-sidecars.yaml b/softclient4es-federation/tests/golden/two-sidecars.yaml index bf1726d..8353c81 100644 --- a/softclient4es-federation/tests/golden/two-sidecars.yaml +++ b/softclient4es-federation/tests/golden/two-sidecars.yaml @@ -612,23 +612,43 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: "python:3.12-slim" command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql==1.6.0 \ + adbc_driver_manager==1.6.0 python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "fed-softclient4es-federation" - conn = dbapi.connect(f"grpc://{host}:32020") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:32020" expected = 2 - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY diff --git a/softclient4es-federation/values.yaml b/softclient4es-federation/values.yaml index fc640bc..07af596 100644 --- a/softclient4es-federation/values.yaml +++ b/softclient4es-federation/values.yaml @@ -115,6 +115,16 @@ license: secretName: "" licenseKeyKey: license-key # Secret data key holding the JWT license -> SOFTCLIENT4ES_LICENSE_KEY apiKeyKey: api-key # Secret data key holding the API key -> SOFTCLIENT4ES_API_KEY + # ── Offline license public-key (Story 16.5, FACT F #2) ──────────────────────── + # OPTIONAL. When the federation must verify a license JWT WITHOUT reaching the + # license server's JWKS endpoint (air-gapped clusters, or a test-signed JWT whose + # `kid` is not in the prod JWKS), point publicKeySecretName at a Secret whose + # `publicKeyKey` data key holds the matching Ed25519 public JWK. It is mounted as + # SOFTCLIENT4ES_LICENSE_PUBLIC_KEY — the air-gap path in LicenseKeyVerifier.loadPublicKey + # (Step 3, VERIFIED). Leave empty (the default) to use the normal JWKS fetch. Renders + # NOTHING when empty, so the golden render and every shipped example are unaffected. + publicKeySecretName: "" + publicKeyKey: license-public-key # Secret data key holding the Ed25519 public JWK -> SOFTCLIENT4ES_LICENSE_PUBLIC_KEY service: type: ClusterIP @@ -313,9 +323,12 @@ sidecarDefaults: drop: - ALL -# `helm test` smoke Job (Story 16.2). Connects to the federation Flight SQL endpoint -# and asserts GetCatalogs (one catalog per registered sidecar) == len(sidecars). +# `helm test` smoke Job (Story 16.2, hardened for CI in Story 16.5). Connects to the +# federation Flight SQL endpoint and asserts GetCatalogs (one catalog per registered +# sidecar) == len(sidecars), with a bounded connect-retry (the federation is NotReady +# until ALL downstreams are discovered — the gRPC readiness aggregate is all-or-nothing). # The default image pip-installs ADBC at runtime (needs PyPI reachability) — pin an -# internal/pre-baked image for air-gapped or rate-limited clusters. +# internal/pre-baked image (test.image) for air-gapped or rate-limited clusters. test: image: "" # default "python:3.12-slim" when empty + adbcVersion: "1.6.0" # adbc_driver_flightsql / adbc_driver_manager version (runtime pip when image not pre-baked) From 6b046e06f677a58e444d0da90a9150bc8bf7da81 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Manciot?= Date: Tue, 23 Jun 2026 10:48:48 +0200 Subject: [PATCH 2/2] =?UTF-8?q?Story=2016.6=20=E2=80=94=20Operator=20guide?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the canonical operator guide (softclient4es-federation/docs/operator-guide.md) that ships with the chart: prerequisites, single-cluster + multi-cluster paths, configuration reference, secret-backend chooser, licensing quota, migration, upgrades/rollback, troubleshooting + SRE incident-triage walkthrough. Authoritative copy — mirror language and private-repo references removed; working-directory convention uses softclient4es-federation/. Same Chart.yaml/README/CI rewrites as prior stories (softclient4es.dev metadata, public DockerHub images, chart-only CI). Closes #12 Refs #11 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../docs/operator-guide.md | 225 ++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 softclient4es-federation/docs/operator-guide.md diff --git a/softclient4es-federation/docs/operator-guide.md b/softclient4es-federation/docs/operator-guide.md new file mode 100644 index 0000000..35e9d91 --- /dev/null +++ b/softclient4es-federation/docs/operator-guide.md @@ -0,0 +1,225 @@ +# SoftClient4ES Federation — Operator Guide + +> **Authoritative guide.** This is the canonical operator guide that ships with the federation Helm chart at `softclient4es-federation/docs/operator-guide.md`. The web and core-docs copies point here. + +The SoftClient4ES federation Helm chart deploys a **cross-cluster Arrow Flight SQL coordinator** (the *federation server*) plus one **per-ES-version sidecar** for each Elasticsearch cluster you want to federate. One `values.yaml` + `helm install` replaces 30+ minutes of hand-written HOCON, Kubernetes manifests, Secrets, and probe wiring. + +## 1. Prerequisites + +- A Kubernetes cluster (v1.27+ recommended for native gRPC readiness probes; on **older clusters set `federation.probes.useGrpc: false`** for TCP readiness — no extra binary needed; see §9/§10). +- **Helm 3**. +- One or more Elasticsearch clusters **reachable from the K8s cluster** (each sidecar opens a connection to exactly one ES cluster of its compiled-against major version). +- A container runtime able to pull from **public DockerHub** (`docker.io/softnetwork/...`). For rate-limit avoidance on first install, configure `imagePullSecrets` with an authenticated DockerHub account. +- (Multi-cluster only) a **Pro or Enterprise license** — see §7. Single-cluster federation runs free on the Community tier. + +> **Image availability:** the federation image `docker.io/softnetwork/softclient4es-federation` is published to public DockerHub at R1 release. The four sidecar images `docker.io/softnetwork/softclient4es{6,7,8,9}-arrow-flight-sql` are already published. Always pin `image.tag` to the published release tag in your `values.yaml`. + +**Working directory convention.** Every `cp examples/…` and `helm install … ./softclient4es-federation` command in this guide is run **from the chart directory `softclient4es-federation/`** (where `Chart.yaml`, `examples/`, and the `./softclient4es-federation` chart path resolve). `cd` there first: + +```bash +cd softclient4es-federation/ # all commands below run from here +``` + +The release name you pass to `helm install …` (this guide uses `fed`) becomes part of the in-cluster Service DNS — `-softclient4es-federation..svc.cluster.local`. Keep it consistent across `install`, `upgrade`, `test`, and `rollback`. + +## 2. The 5-minute path — single cluster + +```bash +# 1. Copy the starter example +cp examples/single-cluster/values.yaml my-values.yaml + +# 2. Edit my-values.yaml — set the one sidecar's ES coordinates + version: +# sidecars[0].elasticsearchVersion: 8 +# sidecars[0].elasticsearch.url: https://es.mycorp.internal:9200 +# sidecars[0].elasticsearch.credentialsSecretName: my-es-secret # see §6 +# image.tag: + +# 3. Create your ES credentials Secret (raw example; §6 for SealedSecrets/ESO/Vault) +kubectl create secret generic my-es-secret \ + --from-literal=es-auth-method=basic \ + --from-literal=es-username=elastic \ + --from-literal=es-password='' + +# 4. Install +helm install fed ./softclient4es-federation -f my-values.yaml + +# 5. Validate +helm test fed # runs GetCatalogs; passes when the federation lists 1 catalog +``` + +Single-cluster is **license-free** (Community `maxClusters=1`). The federation exposes Flight SQL on `ClusterIP` port **32020**; reach it in-cluster at `fed-softclient4es-federation..svc.cluster.local:32020`, or expose it via an Ingress (§6, TLS). + +See [`examples/single-cluster/README.md`](../examples/single-cluster/README.md) for the full single-cluster narrative + topology diagram. + +## 3. Multi-cluster — three regions, mixed ES versions + +```bash +cp examples/three-region/values.yaml my-values.yaml +# Per-region sidecars (us-east-1 ES8 default · eu-west-1 ES8 · ap-south-1 ES9), +# each with its own elasticsearch.credentialsSecretName + auth.credentialsSecretName. + +# 1. Create the Pro/Enterprise license Secret (REQUIRED — 3 clusters > Community's quota of 1; see §7). +# Replace with the JWT from the portal. +kubectl create secret generic sc4es-pro-license \ + --from-literal=license-key='' + +# 2. Create the per-region ES + sidecar-auth Secrets the example references +# (one ES Secret + one sidecar-auth Secret per region; see §6 for SealedSecrets/ESO/Vault). +# Replace every <…> with your real coordinates. +for r in us-east-1 eu-west-1 ap-south-1; do + kubectl create secret generic sc4es-es-$r \ + --from-literal=es-auth-method=basic \ + --from-literal=es-username='' \ + --from-literal=es-password='' + kubectl create secret generic sc4es-arrow-$r \ + --from-literal=arrow-bearer-token='' +done + +# 3. Install (REQUIRES the Pro/Enterprise license Secret above — without it the federation CrashLoops, see §7/§10). +helm install fed ./softclient4es-federation -f my-values.yaml +helm test fed # GetCatalogs → 3 catalogs +``` + +> The example's `values.yaml` references these Secrets by name (`license.secretName: sc4es-pro-license`, each region's `elasticsearch.credentialsSecretName: sc4es-es-` + `auth.credentialsSecretName: sc4es-arrow-`). **Create them BEFORE `helm install`** — a missing Secret is the #1 first-install failure (a loud, self-healing CrashLoop; §6/§10). The exact key names (`license-key`, `es-auth-method`/`es-username`/`es-password`, `arrow-bearer-token`) are the chart's expected Secret data keys (§6). See [`examples/three-region/README.md`](../examples/three-region/README.md) for the full per-region narrative (GDPR data-residency, SRE incident-triage, mixed-version migration). + +Each `sidecars[]` entry deploys one Deployment + one Service and registers one `arrow.flight.federation.servers.` entry (host = `..svc.cluster.local:32010`). Mixed ES versions are first-class — the chart selects `softclient4es{6,7,8,9}-arrow-flight-sql` per `elasticsearchVersion`. **Adding a cluster** = add a `sidecars[]` entry + `helm upgrade`; **removing** = delete the entry + `helm upgrade`. + +> ⚠️ **All-or-nothing readiness (SPOF).** With the default `federation.probes.useGrpc: true`, the federation is Ready only if **every** sidecar's backing ES is reachable — one down region pulls the whole federation out of its Service. For partial availability set `federation.probes.useGrpc: false` (TCP readiness on 32020); the federation then stays Ready and fails only the queries that touch the down region. See §9. + +## 4. The R2b-ready path + +```bash +cp examples/heterogeneous-ready/values.yaml my-values.yaml +``` + +This is the three-region topology **plus commented-out R2b `duckdb-attach` placeholders** (PostgreSQL / MySQL / Snowflake) that show how heterogeneous sources slot into the same `servers` map when R2b ships (Epic 25/26). **In R1 they stay commented** — `GetCatalogs` returns 3, not 6. R1 has no `values.yaml` key for `duckdb-attach` (the `servers` map is ConfigMap-rendered from `sidecars[]` Flight SQL servers only). **Uncommenting them later counts toward the license quota** (`clusterCount` includes attach servers) — 3 ES + 3 attach = 6 > Pro's 5 → Enterprise. + +See [`examples/heterogeneous-ready/README.md`](../examples/heterogeneous-ready/README.md) for the R2b preview narrative. + +## 5. Configuration reference + +> `values.yaml` field → HOCON path / env var. **Verify against the chart's own `values.yaml` and README; this table is the authoritative mapping.** + +**Federation** (image `softclient4es-federation`; HOCON root `arrow.flight.federation.*`): + +| `values.yaml` | HOCON | Env var | Default | +|---|---|---|---| +| `replicaCount` | — | — | 1 | +| `image.repository` / `image.tag` / `image.pullPolicy` | — | — | `docker.io/softnetwork/softclient4es-federation` / `appVersion` / `IfNotPresent` | +| `federation.maxMemory` | `max-memory` | `FEDERATION_MAX_MEMORY` | `512m` | +| `federation.queryTimeoutSeconds` | `query-timeout-seconds` | `FEDERATION_QUERY_TIMEOUT` | 30 | +| `federation.health.port` | `health.port` | `FEDERATION_HEALTH_PORT` | 32021 | +| `federation.health.probeTimeoutSeconds` | `health.probe-timeout-seconds` | `FEDERATION_HEALTH_PROBE_TIMEOUT` | 5 | +| `federation.duckdb.path` | `duckdb.path` | `FEDERATION_DUCKDB_PATH` | `:memory:` | +| `federation.upgradeUrl` | `upgrade-url` | `FEDERATION_UPGRADE_URL` | portal pricing URL | +| (`host`/`port` fixed by chart) | `host` / `port` | `FEDERATION_HOST` / `FEDERATION_PORT` | `0.0.0.0` / 32020 | +| `federation.probes.useGrpc` | — (probe wiring) | — | true | +| `federation.tls.{enabled,secretName}` | — (Ingress) | — | false | +| `ingress.{enabled,className,annotations,hosts,tls}` | — (Ingress) | — | false | +| `license.publicKeySecretName` (+ `license.publicKeyKey`) | — | `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY` | `""` / `license-public-key` | + +**Sidecar** (per `sidecars[]`; image auto-selected by `elasticsearchVersion`; HOCON root `arrow.flight.*` + elasticsql core `elastic.credentials.*`): + +| `values.yaml` | HOCON / config | Env var | Default | +|---|---|---|---| +| `sidecars[].elasticsearchVersion` (`6\|7\|8\|9`) | — (image selection) | — | — | +| `sidecars[].name` | `servers.` key (RFC1123, unique) | — | — | +| `sidecars[].elasticsearch.url` (or `.scheme/.host/.port`) | `elastic.credentials.*` | `ELASTIC_SCHEME` / `ELASTIC_HOST` / `ELASTIC_PORT` | — (NO single ES-URL env) | +| `sidecars[].elasticsearch.credentialsSecretName` | `elastic.credentials.*` | `ELASTIC_AUTH_METHOD` / `ELASTIC_CREDENTIALS_{USERNAME,PASSWORD,API_KEY,BEARER_TOKEN}` | — | +| `sidecars[].arrow.batchSize` | `batch-size` | `ARROW_BATCH_SIZE` | 1000 | +| `sidecars[].arrow.queryTimeoutSeconds` | `query-timeout-seconds` | `ARROW_QUERY_TIMEOUT_SECONDS` | 120 | +| `sidecars[].arrow.maxMemory` | `join.max-memory` | `ARROW_JOIN_MAX_MEMORY` | `256m` | +| `sidecars[].auth.{method,credentialsSecretName}` | `auth.*` + federation `servers..credentials` | `ARROW_AUTH_{METHOD,USERNAME,PASSWORD,BEARER_TOKEN,API_KEY}` + `CONFIG_FORCE_*` | `none` | +| `sidecars[].default` | `servers..default` | — | false (≤1 true) | +| `sidecars[].alias` | `servers..alias` | — | `name` | +| `sidecars[].tls` | `servers..tls` (outgoing) | — | false | +| `sidecars[].replicaCount` / `.resources` / `.image.{repository,tag}` | — | — | 1 / — / version-default | + +**License & telemetry** (elasticsql `licensing`; HOCON root `softclient4es.*`): + +| `values.yaml` | HOCON | Env var | Default | +|---|---|---|---| +| `license.secretName` → `license-key` | `softclient4es.license.key` | `SOFTCLIENT4ES_LICENSE_KEY` | `""` (Community) | +| `license.secretName` → `api-key` | `softclient4es.license.api-key` | `SOFTCLIENT4ES_API_KEY` | `""` | +| `telemetry.enabled` | `softclient4es.telemetry.enabled` | `SOFTCLIENT4ES_TELEMETRY_ENABLED` | true | + +> **No env override** (NOT chart-exposed): `softclient4es.license.{connect-timeout,read-timeout,grace-period,cache-dir,refresh.enabled,refresh.interval}` and `softclient4es.license.telemetry.enabled` (the license-refresh-metrics switch — DISTINCT from the daily-ping `softclient4es.telemetry.enabled` above; do not confuse them). + +## 6. Secret management — choosing a backend + +The chart **never creates a `Secret`** — you create it (raw, SealedSecrets, External Secrets Operator, or Vault Agent Injector) and reference it by name. One Secret feeds **both** the sidecar (`ARROW_AUTH_*`, `ELASTIC_*`) and, for federation→sidecar auth, the federation (`CONFIG_FORCE_*` via Typesafe Config `override_with_env_vars`). The four backends are documented in depth in [`docs/secret-backends.md`](secret-backends.md), with ready-to-adapt manifests under [`examples/sealed-secrets/`](../examples/sealed-secrets/) and [`examples/external-secrets/`](../examples/external-secrets/). In short: + +- **Raw `kubectl create secret`** — simplest; fine for dev / GitOps with encrypted-at-rest etcd. Keys: `es-auth-method`/`es-username`/`es-password`/`es-api-key`/`es-bearer-token` (ES), `arrow-username`/`arrow-password`/`arrow-bearer-token`/`arrow-api-key` (sidecar auth), `license-key`/`api-key` (license), `tls.crt`/`tls.key` (TLS). +- **SealedSecrets** — commit an encrypted `SealedSecret` to Git; the in-cluster controller decrypts it. Re-seal per controller (certs are per-controller; `kubeseal` CLI version must match the controller `appVersion`, not the Helm chart version). +- **External Secrets Operator (ESO)** — sync from AWS/GCP Secret Manager or Vault. Examples use `external-secrets.io/v1` (the stable API; `v1beta1` was removed at ESO v0.17). +- **Vault Agent Injector** — inject secrets as files/env via pod annotations. + +> **RFC1123 sidecar names are mandatory.** The federation→sidecar credential injection mangles the sidecar `name` into a `CONFIG_FORCE_*` env path; a name with `_`/`.`/uppercase mangles wrong and the federation `sys.exit(1)`s. Use `[a-z0-9-]` names. **A wrong/missing Secret is a loud, SELF-HEALING CrashLoop** — the pod restarts and boots Ready once the Secret materializes (e.g. after ESO syncs). Do NOT uninstall; check the Secret's data keys. + +**TLS at the Ingress, not the pod.** The federation listener is plaintext-only (`Location.forGrpcInsecure`). `federation.tls.{enabled,secretName}` drives an **Ingress** `tls:` block (cert-manager `kubernetes.io/tls`); since Flight SQL is gRPC, the Ingress controller must proxy gRPC — nginx annotation `nginx.ingress.kubernetes.io/backend-protocol: "GRPC"` (NOT `GRPCS`; the pod upstream is plaintext h2c). This is DISTINCT from `sidecars[].tls` (outgoing per-downstream client TLS → `servers..tls`). In-cluster plaintext clients hit the Service (32020) directly. + +## 7. Licensing + +The federation reads its license from a referenced Secret as `SOFTCLIENT4ES_LICENSE_KEY` (offline Ed25519 JWT) and/or `SOFTCLIENT4ES_API_KEY` (automated portal provisioning). The gate is a **per-platform `maxClusters` quota**, enforced at startup against the number of `servers` (sidecars): + +| Tier | `maxClusters` | Federation clusters you can run | +|---|---|---| +| **Community** (no license) | 1 | single-cluster federation — **free** | +| **Pro** | 5 | up to 5 sidecars / clusters | +| **Enterprise** | unlimited | any | + +Exceeding the quota → the federation logs the over-quota error and `sys.exit(1)` → **CrashLoopBackOff by design** (see §10). Federation is NOT a paid feature — single-cluster is the free adoption tier; the quota is on *cluster count*. To opt out of the daily anonymous usage ping, set `telemetry.enabled: false` (→ `SOFTCLIENT4ES_TELEMETRY_ENABLED=false`); this has zero impact on functionality or your license. + +> **Offline verification.** For air-gapped or strict-egress clusters, mount the Ed25519 public JWK via `license.publicKeySecretName` (key `license.publicKeyKey`, default `license-public-key`) → `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY`. This lets the federation verify a Pro/Enterprise JWT entirely offline (no portal round-trip). It is gated independently of `license.secretName`. + +## 8. ES-version mixing & migration + +Sidecars are independent — there is **no constraint** on mixing ES 6/7/8/9 in one federation at R1. The canonical migration story: run ES 8 sidecars in two regions while you migrate one region to ES 9; flip that region's `sidecars[].elasticsearchVersion` from `8` to `9` and `helm upgrade` once the ES cluster is upgraded. Cross-version JOINs with version-specific SQL features are the one caveat — flag them in testing (advanced cross-version cases surface in customer testing). + +## 9. Upgrades & rollback + +- **Upgrade:** edit `values.yaml`, `helm upgrade fed ./softclient4es-federation -f my-values.yaml`. Adding/removing a sidecar re-renders the federation ConfigMap; the federation pod restarts to pick up the new `servers` map. +- **Rollback:** `helm rollback fed ` (`helm history fed` to list). +- **Secret rotation** is start-time: after changing a Secret, `kubectl rollout restart deployment` for BOTH the sidecar AND the federation (one Secret feeds both — restart both or the two sides drift). +- **Probes:** native gRPC readiness needs K8s 1.27+. On clusters **older than 1.27**, set `federation.probes.useGrpc: false` (TCP readiness on 32020) — this is the recommended fallback and needs **no extra binary**. The `exec`+`grpc_health_probe` approach is NOT a drop-in: the `grpc_health_probe` binary is **not bundled in the federation image**, so you would have to bake it into a custom image (or an init-container copy) yourself before an `exec` probe can run it. For nearly all operators, `useGrpc: false` is the simpler and supported path below 1.27. + +## 10. Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| Federation pod `CrashLoopBackOff`, log "exceeds maxClusters" / `sys.exit(1)` | More sidecars than the license tier allows (Community=1, Pro=5) | Add/upgrade the license Secret (Pro/Enterprise), or reduce `sidecars[]`. §7 | +| `CrashLoopBackOff` after uncommenting R2b `duckdb-attach` | `clusterCount` counts attach servers too → over Pro's 5 | Enterprise license, or keep R2b commented in R1. §4 | +| `CrashLoopBackOff`, "invalid credentials" at config load | `auth.method` ≠ none with creds only in a Secret but `federation.credentialsFromEnv` off, OR a non-RFC1123 sidecar name mangled the `CONFIG_FORCE_*` path | Enable `credentialsFromEnv`; rename sidecar to `[a-z0-9-]`. §6 | +| `CrashLoopBackOff` immediately at boot, no clear error | `readOnlyRootFilesystem` with no writable `/tmp` — the DuckDB JNI native lib can't extract | Ensure the `/tmp` `emptyDir` is present (chart default); don't override the volume. | +| Federation NotReady, "could not connect to sidecar X" | Sidecar pod / its backing ES unreachable; all-or-nothing gRPC readiness pulls the whole federation | Check sidecar pod + ES; for partial availability set `federation.probes.useGrpc: false`. §3/§9 | +| Wrong Secret keys / ESO sync lag | `optional:true` env can't validate Secret CONTENTS → boot-time `validate()` CrashLoop | Fix the Secret's data keys; the pod self-heals on the next restart — do NOT uninstall. §6 | +| License validation failure (**Pro/Enterprise only**) | A license JWT WAS supplied but is malformed / clock-skewed / expired beyond the 14-day grace | Re-provision via the portal (`SOFTCLIENT4ES_API_KEY`); check the JWT and node clock. **N.B.** with NO license (blank `SOFTCLIENT4ES_LICENSE_KEY`) the federation does NOT error — it runs on Community (1-cluster quota); that fallback is expected, not a failure. Community has no API-key/refresh path, so the portal re-provision step applies ONLY to a paid license. | +| TLS cert renewal failure | Cert lives at the **Ingress** (federation pod is plaintext) | Renew the cert-manager Certificate / Ingress `tls` Secret; nginx needs `backend-protocol: "GRPC"`. §6 (TLS) | +| ES connection failure from a sidecar | Wrong `ELASTIC_SCHEME/HOST/PORT/AUTH_METHOD` or ES creds | Verify the ES Secret + reachability; remember there is NO single ES-URL — scheme/host/port are separate. §5 | +| ES-version mismatch | `elasticsearchVersion` doesn't match the backing ES major | Set the sidecar's `elasticsearchVersion` to the ES cluster's major. §8 | + +### SRE incident-triage walkthrough (three-region) + +1. **Alert:** dashboards/queries failing across regions. `kubectl get pods` → the `fed-softclient4es-federation` pod is `0/1 NotReady`. +2. **Confirm the SPOF:** `kubectl describe pod` shows the gRPC readiness probe failing. With `useGrpc:true`, one unreachable region makes the *whole* federation NotReady (all-or-nothing aggregate). +3. **Find the bad region:** `kubectl get pods -l app.kubernetes.io/component=sidecar` → identify the `NotReady` sidecar; `kubectl logs` it for the ES connection error. +4. **Mitigate fast:** re-apply your existing values file AND flip the probe — `helm upgrade fed ./softclient4es-federation -f my-values.yaml --set federation.probes.useGrpc=false` → the federation goes Ready and serves the two healthy regions; only queries touching the down region fail. (Per-alias readiness is an R1.x backlog item.) ⚠️ **Always include `-f my-values.yaml`**: a bare `helm upgrade … --set …` resets every other value to the chart default (Helm does NOT reuse the previous release's values unless you pass `-f` or `--reuse-values`), which would silently revert your three-region `sidecars[]` / license / topology mid-incident. +5. **Fix root cause:** restore the down region's ES / sidecar, then flip `useGrpc` back to `true` for fail-closed semantics. + +## 11. Performance tuning *(placeholder — iterate after R1 telemetry)* + +> Hard numbers land after R1 telemetry. The qualitative levers today: + +- **Scale federation replicas** (`replicaCount`) for concurrent-query throughput; the federation is stateless per query (DuckDB `:memory:`). +- **Scale sidecars** (`sidecars[].replicaCount`, document HA at 3) for per-cluster availability and parallelism. +- **`federation.maxMemory` / `sidecars[].arrow.maxMemory`** size the DuckDB / JOIN engine; raise for large cross-cluster JOINs, watch pod memory limits. +- **`batchSize`** trades latency vs throughput on streaming. +- Resource recommendations per cluster-count tier: TBD — seeded after R1 telemetry. + +## 12. SLA implications + +Pro includes 48h support. This guide is intended to be authoritative enough that the common scenarios (install, add/remove a cluster, mixed-version migration, the over-quota and connectivity failure modes) are self-serviceable without a ticket. + +## For Epic 17 + +Epic 17 (Documentation & Marketing) consumes, for R1 launch-day docs: **this operator guide** (linked/included from the launch docs), the **`examples/` topology folders** ([single-cluster](../examples/single-cluster/README.md), [three-region](../examples/three-region/README.md), [heterogeneous-ready](../examples/heterogeneous-ready/README.md) — each with README + ASCII diagram), and the **SRE incident-triage walkthrough** (§10, linked from the R1 marketing SRE-wedge story).