From ebdf6bcf04a43dc76d255e5440f2b57070b401b4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Manciot?= Date: Tue, 23 Jun 2026 10:27:39 +0200 Subject: [PATCH 1/3] =?UTF-8?q?Story=2016.4=20=E2=80=94=20Topology=20examp?= =?UTF-8?q?les?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add single-cluster / three-region / heterogeneous-ready example overlays (each with README + ASCII diagram + install command + SHOW CATALOGS expectation) and their per-example goldens. three-region and heterogeneous-ready renders are byte-identical by design; the duckdb-attach source marker (3 vs 0) is the only discriminator. Same path/repo rewrites as prior stories. Closes #7 Co-Authored-By: Claude Opus 4.8 (1M context) --- softclient4es-federation/README.md | 36 +- .../examples/heterogeneous-ready/README.md | 74 ++ .../examples/heterogeneous-ready/values.yaml | 108 ++ .../examples/single-cluster/README.md | 70 ++ .../examples/single-cluster/values.yaml | 48 + .../examples/three-region/README.md | 107 ++ .../examples/three-region/values.yaml | 63 ++ .../golden/example-heterogeneous-ready.yaml | 948 ++++++++++++++++++ .../tests/golden/example-single-cluster.yaml | 421 ++++++++ .../tests/golden/example-three-region.yaml | 948 ++++++++++++++++++ 10 files changed, 2821 insertions(+), 2 deletions(-) create mode 100644 softclient4es-federation/examples/heterogeneous-ready/README.md create mode 100644 softclient4es-federation/examples/heterogeneous-ready/values.yaml create mode 100644 softclient4es-federation/examples/single-cluster/README.md create mode 100644 softclient4es-federation/examples/single-cluster/values.yaml create mode 100644 softclient4es-federation/examples/three-region/README.md create mode 100644 softclient4es-federation/examples/three-region/values.yaml create mode 100644 softclient4es-federation/tests/golden/example-heterogeneous-ready.yaml create mode 100644 softclient4es-federation/tests/golden/example-single-cluster.yaml create mode 100644 softclient4es-federation/tests/golden/example-three-region.yaml diff --git a/softclient4es-federation/README.md b/softclient4es-federation/README.md index 96a78db..6c7c234 100644 --- a/softclient4es-federation/README.md +++ b/softclient4es-federation/README.md @@ -267,8 +267,8 @@ The committed render baselines under `tests/golden/` let a future PR detect temp drift: `default.yaml` (0 sidecars), `two-sidecars.yaml` (federation + 2 mixed ES8/ES9 sidecars), `secret-auth.yaml` (1 sidecar, Secret-backed ES + sidecar bearer auth + federation `CONFIG_FORCE_*`), and `ingress-tls.yaml` (federation behind a cert-manager -TLS Ingress). Regenerate (and -review the diff) with: +TLS Ingress). The `example-*.yaml` baselines render the three topology examples under +`examples/` (Story 16.4). Regenerate (and review the diff) with: ```sh helm template fed ./softclient4es-federation > ./softclient4es-federation/tests/golden/default.yaml @@ -278,6 +278,13 @@ helm template fed ./softclient4es-federation -f ./softclient4es-federation/tests > ./softclient4es-federation/tests/golden/secret-auth.yaml helm template fed ./softclient4es-federation -f ./softclient4es-federation/tests/values/ingress-tls.yaml \ > ./softclient4es-federation/tests/golden/ingress-tls.yaml +# Topology examples (Story 16.4): +helm template fed ./softclient4es-federation -f ./softclient4es-federation/examples/single-cluster/values.yaml \ + > ./softclient4es-federation/tests/golden/example-single-cluster.yaml +helm template fed ./softclient4es-federation -f ./softclient4es-federation/examples/three-region/values.yaml \ + > ./softclient4es-federation/tests/golden/example-three-region.yaml +helm template fed ./softclient4es-federation -f ./softclient4es-federation/examples/heterogeneous-ready/values.yaml \ + > ./softclient4es-federation/tests/golden/example-heterogeneous-ready.yaml git diff --stat ./softclient4es-federation/tests/golden/ ``` @@ -285,3 +292,28 @@ Any diff must be intentional. CI (Story 16.5) enforces these goldens plus `helm and `kubeconform -strict` (including the 2-sidecar, Secret-backed, and TLS/Ingress renders). The chart `templates/` must emit ZERO `kind: Secret` — assert with `helm template fed ./softclient4es-federation -f … | grep -c '^kind: Secret'` (expected `0`). + +> **`example-three-region.yaml` and `example-heterogeneous-ready.yaml` are BYTE-IDENTICAL +> by design** — `diff` between them is EMPTY (Story 16.4 Decision A2). The two `values.yaml` +> overlays carry the SAME active values (same license Secret, telemetry, `useGrpc`, and the +> same three `sidecars[]`); they differ ONLY in comments and the commented-out R2b +> `duckdb-attach` preview, and Helm strips all comments before rendering. The ONLY signal +> distinguishing the two examples is the SOURCE: `examples/heterogeneous-ready/values.yaml` +> contains the `type = "duckdb-attach"` R2b marker and `examples/three-region/values.yaml` +> does not (Decision A2b) — CI (16.5) greps for this. A regression that overwrote one +> overlay with the other would pass the golden + `helm test` gates but FAIL the grep. + +## Topology examples + +Copy-pasteable `values.yaml` overlays for three operator scenarios live under `examples/` +(Story 16.4). Each has its own `README.md` with a "when to use", an ASCII topology diagram, +the install command, and the `SHOW CATALOGS` smoke expectation: + +| Example | Topology | License | `SHOW CATALOGS` | +|---|---|---|---| +| `examples/single-cluster/` | federation + 1 ES8 sidecar | **none** (Community, `maxClusters=1`) | 1 | +| `examples/three-region/` | federation + us-east-1 (ES8) / eu-west-1 (ES8) / ap-south-1 (ES9) | **Pro / Enterprise** | 3 | +| `examples/heterogeneous-ready/` | three-region today + a commented R2b `duckdb-attach` preview (PG/MySQL/Snowflake) | **Pro / Enterprise** | 3 (R2b inactive) | + +Install any of them with `helm install softclient4es-federation softclient4es-federation -f examples//values.yaml` +(create the referenced Secrets first — see each example's README). diff --git a/softclient4es-federation/examples/heterogeneous-ready/README.md b/softclient4es-federation/examples/heterogeneous-ready/README.md new file mode 100644 index 0000000..9bbb767 --- /dev/null +++ b/softclient4es-federation/examples/heterogeneous-ready/README.md @@ -0,0 +1,74 @@ +# Heterogeneous-ready federation (R1 today → R2b tomorrow) — Pro / Enterprise + +Runs the **three-region ES topology TODAY on R1** (3 Elasticsearch sidecars), and is +laid out so the **R2b heterogeneous sources** (PostgreSQL, MySQL, Snowflake, …) slot +into the **same federation `servers` map** when R2b ships — **no restructuring** of +this deployment. + +## What works today (R1) + +Exactly the three-region topology: a Flight SQL endpoint federating 3 ES clusters +(us-east-1 ES 8, eu-west-1 ES 8, ap-south-1 ES 9), with cross-cluster JOIN. Needs a +**Pro/Enterprise license** (3 clusters > Community's `maxClusters=1`) — see the +quota table in `../three-region/README.md`. + +## What slots in tomorrow (R2b) + +The federation `servers` map already supports `type = "duckdb-attach"` entries +(PostgreSQL / MySQL / Snowflake / … via DuckDB ATTACH). When R2b ships, those join +the SAME map your ES sidecars register into — a relational source becomes a federated +catalog you can `JOIN` against your ES indices, with no change to the federation +deployment topology. The `values.yaml` carries a commented preview of the exact HOCON. + +> ⚠️⚠️ **The R2b preview in `values.yaml` is INERT on R1 — uncommenting it does NOTHING.** +> R1 has **no `values.yaml` key** that parses `duckdb-attach` (or a raw `servers { }` +> HOCON map). The federation `servers` map is rendered into a ConfigMap **entirely from +> the `sidecars[]` array** (Flight SQL sidecars only). If you uncomment the preview block +> and `helm install`, Helm discards it as comments, the chart ignores it, no relational +> catalog appears, and `SHOW CATALOGS` still returns **3** (the ES sidecars). The block is +> a documentation illustration of the FUTURE R2b shape — not an activation switch. R2b +> (Epic 25/26) is what makes these sources live. + +> ⚠️ **Quota note:** each active `duckdb-attach` server ALSO counts toward `maxClusters` +> (`clusterCount = servers.size`). 3 ES + 3 attach = 6 clusters would exceed Pro's +> `maxClusters=5` → you'd need Enterprise. (A smaller mix still fits Pro: 3 ES + 2 attach +> = 5 clusters is exactly Pro's cap; only the 6th server tips you into Enterprise.) Plan +> your tier before activating R2b sources. + +## Topology + +``` + ┌──────────────────────────────────┐ + Flight SQL client ───┼─► Federation (Flight SQL :32020) │ + └───┬────────┬────────┬─────────────┘ + R1-ACTIVE │ │ │ + ┌─────────────▼┐ ┌─────▼──────┐ ┌▼────────────┐ + │ us-east-1 ES8│ │ eu-west-1 │ │ ap-south-1 │ + │ sidecar │ │ ES8 sidecar│ │ ES9 sidecar │ + └──────────────┘ └────────────┘ └─────────────┘ + + R2b-PREVIEW (commented; slot into the SAME servers map) + ┌╌╌╌╌╌╌╌╌╌╌╌╌╌┐ ┌╌╌╌╌╌╌╌╌╌╌╌╌┐ ┌╌╌╌╌╌╌╌╌╌╌╌╌╌┐ + ┊ analytics_pg ┊ ┊ orders_mysql┊ ┊ snowflake_wh ┊ + ┊ (PostgreSQL) ┊ ┊ (MySQL) ┊ ┊ (Snowflake) ┊ + └╌╌╌╌╌╌╌╌╌╌╌╌╌┘ └╌╌╌╌╌╌╌╌╌╌╌╌┘ └╌╌╌╌╌╌╌╌╌╌╌╌╌┘ + (duckdb-attach — Epic 25/26) +``` + +## Install (R1) + +Same as three-region (Pro license + per-region Secrets), then: + +```bash +helm install softclient4es-federation softclient4es-federation \ + -f examples/heterogeneous-ready/values.yaml +``` + +## Verify + +```bash +helm test softclient4es-federation # SHOW CATALOGS returns 3 (the active ES clusters) +``` + +The R2b `duckdb-attach` placeholders are **commented out** — they do not register +and do not count toward `SHOW CATALOGS` (which returns **3**) until R2b activates them. diff --git a/softclient4es-federation/examples/heterogeneous-ready/values.yaml b/softclient4es-federation/examples/heterogeneous-ready/values.yaml new file mode 100644 index 0000000..a02913c --- /dev/null +++ b/softclient4es-federation/examples/heterogeneous-ready/values.yaml @@ -0,0 +1,108 @@ +# ───────────────────────────────────────────────────────────────────────────── +# SoftClient4ES Federation — HETEROGENEOUS-READY example. +# +# This is the THREE-REGION topology (3 ES sidecars) you can run on R1 TODAY, plus a +# commented-out preview of the R2b "duckdb-attach" heterogeneous sources (PostgreSQL, +# MySQL, Snowflake). When R2b ships, those slot into the SAME federation `servers` +# map WITHOUT restructuring this deployment — the federation already federates them +# the moment they are activated. +# +# ⚠️ Like three-region, this needs a PRO/ENTERPRISE license (3 active ES sidecars = +# 3 clusters; Community maxClusters=1). See README. +# +# Install: +# helm install softclient4es-federation softclient4es-federation \ +# -f examples/heterogeneous-ready/values.yaml +# ───────────────────────────────────────────────────────────────────────────── + +license: + secretName: "sc4es-pro-license" + +telemetry: + enabled: true + +federation: + probes: + useGrpc: true + +# R1-ACTIVE: 3 Elasticsearch sidecars (same as three-region). +sidecars: + - name: us-east-1 + elasticsearchVersion: 8 + elasticsearch: + url: "https://es-us-east-1.example.com:9200" + credentialsSecretName: "sc4es-es-us-east-1" + auth: + method: bearer + credentialsSecretName: "sc4es-arrow-us-east-1" + default: true + - name: eu-west-1 + elasticsearchVersion: 8 + elasticsearch: + url: "https://es-eu-west-1.example.com:9200" + credentialsSecretName: "sc4es-es-eu-west-1" + auth: + method: bearer + credentialsSecretName: "sc4es-arrow-eu-west-1" + - name: ap-south-1 + elasticsearchVersion: 9 + elasticsearch: + url: "https://es-ap-south-1.example.com:9200" + credentialsSecretName: "sc4es-es-ap-south-1" + auth: + method: bearer + credentialsSecretName: "sc4es-arrow-ap-south-1" + +# ───────────────────────────────────────────────────────────────────────────── +# R2b PREVIEW — heterogeneous "duckdb-attach" sources (NOT active in R1). +# +# ⚠️⚠️ UNCOMMENTING THE BLOCK BELOW DOES NOTHING ON R1. ⚠️⚠️ +# This is NOT a chart values key. There is NO `values.yaml` key in R1 that parses +# `duckdb-attach` / a raw `servers { }` HOCON map. The federation `servers` map is +# rendered into a ConfigMap ENTIRELY from the `sidecars[]` array above (Flight SQL +# servers only — one entry per `elasticsearchVersion`). If you uncomment the HOCON +# below and `helm install`, Helm treats it as inert YAML comments, the chart ignores +# it, the federation never sees a PostgreSQL/MySQL/Snowflake catalog, and +# `SHOW CATALOGS` still returns 3 (the ES sidecars only). Do NOT file a "my PG catalog +# didn't show up" ticket — R2b (Epic 25/26) is what wires these into the `servers` map. +# +# The entries below show the EXACT HOCON shape these sources will take in the federation +# `servers` map once R2b ships — copied verbatim from +# federation/src/main/resources/reference.conf. They are illustrative ONLY here. +# +# ⚠️ Each duckdb-attach server, once ACTIVE, counts toward maxClusters too +# (clusterCount = servers.size, ALL server types). 3 ES + 3 attach = 6 clusters → +# exceeds Pro's maxClusters=5 → needs Enterprise. (3 ES + 2 attach = 5 still fits +# Pro exactly; only the 6th server tips into Enterprise.) Plan tier accordingly. +# +# ── DuckDB ATTACH (PostgreSQL) ── +# analytics_pg { +# host = "pg.internal" +# port = 5432 +# type = "duckdb-attach" +# dialect = "postgresql" +# read-only = true +# expected-catalog = "analytics_db" +# credentials { username = "readonly" password = "secret" } +# } +# ── DuckDB ATTACH (MySQL) ── +# orders_mysql { +# host = "mysql.internal" +# port = 3306 +# type = "duckdb-attach" +# dialect = "mysql" +# expected-catalog = "orders_db" +# credentials { username = "reader" password = "secret" } +# } +# ── DuckDB ATTACH (Snowflake — custom attach-string + secret) ── +# snowflake_wh { +# host = "" +# type = "duckdb-attach" +# dialect = "snowflake" +# read-only = true +# duckdb { +# attach-string = "account=my-account" +# secret { TYPE = "snowflake" ACCOUNT = "my-account" USER = "admin" PASSWORD = "pw" } +# } +# } +# ───────────────────────────────────────────────────────────────────────────── diff --git a/softclient4es-federation/examples/single-cluster/README.md b/softclient4es-federation/examples/single-cluster/README.md new file mode 100644 index 0000000..2614d8f --- /dev/null +++ b/softclient4es-federation/examples/single-cluster/README.md @@ -0,0 +1,70 @@ +# Single-cluster federation (FREE tier) + +Federation Flight SQL server in front of **one** Elasticsearch cluster, via one +Arrow Flight SQL sidecar. Runs on the **Community license — no license key needed** +(the cluster quota is `maxClusters=1`, and one sidecar fits). + +## When to use this + +- You have ONE Elasticsearch cluster and want a **Flight SQL endpoint** for it + (BI tools, ADBC, JDBC-over-Flight) with **cross-index JOIN routing** and licensing, + exposed externally — without standing up the federation + sidecar HOCON + probes by hand. +- You want to **start free** and grow into multi-region later: add a second `sidecars[]` + entry + a Pro license and you have the three-region topology (see `../three-region/`). + +## Topology + +``` + ┌─────────────────────────────┐ + Flight SQL client │ Federation (Flight SQL) │ + (BI / ADBC / JDBC) ─┼─► :32020 softclient4es- │ + │ federation │ + │ health gRPC :32021 │ + └──────────────┬──────────────┘ + │ in-cluster gRPC (plaintext) + ▼ + ┌─────────────────────────────┐ + │ Sidecar: primary (ES 8) │ + │ Arrow Flight SQL :32010 │ + └──────────────┬──────────────┘ + │ ELASTIC_SCHEME/HOST/PORT + ▼ + ┌─────────────────────────────┐ + │ Elasticsearch (1 cluster) │ + │ es.example.com:9200 │ + └─────────────────────────────┘ +``` + +## Install + +```bash +# Create the ES credentials Secret (skip for a noauth ES): +kubectl create secret generic sc4es-es-credentials \ + --from-literal=es-auth-method=basic \ + --from-literal=es-username=elastic \ + --from-literal=es-password='' + +helm install softclient4es-federation softclient4es-federation \ + -f examples/single-cluster/values.yaml \ + --set sidecars[0].elasticsearch.url=https://YOUR-ES:9200 +``` + +> **`elasticsearch.url` must be `scheme://host:port`.** The chart decomposes it into +> `ELASTIC_SCHEME`/`ELASTIC_HOST`/`ELASTIC_PORT` (there is no single ES-URL env). A +> scheme-less value defaults to `http`; a port-less value defaults to `9200`. For a +> TLS or non-9200 cluster, always include `https://` and the explicit port, or set +> `elasticsearch.scheme`/`.host`/`.port` directly (explicit fields win over `url`). + +## Verify + +```bash +helm test softclient4es-federation # SHOW CATALOGS returns 1 (the `primary` cluster) +``` + +`SHOW CATALOGS` (or ADBC `get_objects(depth="catalogs")`) returns **1** catalog. + +## Licensing + +**No license required.** A single-cluster federation runs on Community +(`maxClusters=1`). The moment you add a **second** sidecar you need a Pro/Enterprise +license — see `../three-region/README.md`. diff --git a/softclient4es-federation/examples/single-cluster/values.yaml b/softclient4es-federation/examples/single-cluster/values.yaml new file mode 100644 index 0000000..03a46c9 --- /dev/null +++ b/softclient4es-federation/examples/single-cluster/values.yaml @@ -0,0 +1,48 @@ +# ───────────────────────────────────────────────────────────────────────────── +# SoftClient4ES Federation — SINGLE-CLUSTER example. +# +# Federation Flight SQL server + ONE Arrow Flight SQL sidecar in front of ONE +# Elasticsearch cluster. This is the FREE tier: a single sidecar boots on the +# Community license (no license.secretName needed — maxClusters=1 permits exactly +# one cluster). Use it to expose JOIN routing + licensing + a Flight SQL endpoint +# for ONE ES cluster, externally reachable, with zero licensing cost. +# +# Install: +# helm install softclient4es-federation softclient4es-federation \ +# -f examples/single-cluster/values.yaml +# +# EDIT BEFORE INSTALL: sidecars[0].elasticsearch.url and (if your ES needs auth) +# create a Secret named `sc4es-es-credentials` and reference it below. +# ───────────────────────────────────────────────────────────────────────────── + +# No license needed — a single-cluster federation runs on Community (maxClusters=1). +license: + secretName: "" + +# Daily product-instance telemetry ping is ON by default. Set false to opt out. +telemetry: + enabled: true + +# Single downstream sidecar (no SPOF amplification → gRPC readiness default is fine). +sidecars: + - name: primary # RFC1123; servers.primary + k8s suffix + elasticsearchVersion: 8 # 6|7|8|9 → selects the sidecar image + elasticsearch: + # REPLACE with your ES endpoint. The chart DECOMPOSES this into + # ELASTIC_SCHEME/ELASTIC_HOST/ELASTIC_PORT (there is NO single ES-URL env). + # ⚠️ MUST be the canonical `scheme://host:port` form. A scheme-less value + # silently defaults to scheme=http; a port-less value silently defaults to + # port=9200 — either can break the ES connect against a TLS/non-9200 cluster. + # If your URL has a path/userinfo, set explicit scheme/host/port instead. + url: "https://es.example.com:9200" + # scheme: https # optional explicit override (wins over url) -> ELASTIC_SCHEME + # host: es.example.com # optional explicit override (wins over url) -> ELASTIC_HOST + # port: 9200 # optional explicit override (wins over url) -> ELASTIC_PORT + # K8s Secret you create (chart does NOT create it). Keys: es-auth-method, + # es-username, es-password, es-api-key, es-bearer-token. Leave empty for noauth ES. + credentialsSecretName: "sc4es-es-credentials" + default: true # the only cluster → bare-table routing + # Sidecar incoming auth OFF (intra-cluster trust). Federation reaches it plaintext + # over the in-cluster Service. For a Secret-backed sidecar auth, see 16.3 docs. + auth: + method: none diff --git a/softclient4es-federation/examples/three-region/README.md b/softclient4es-federation/examples/three-region/README.md new file mode 100644 index 0000000..caaf28c --- /dev/null +++ b/softclient4es-federation/examples/three-region/README.md @@ -0,0 +1,107 @@ +# Three-region federation (mixed ES versions) — Pro / Enterprise + +Federation + 3 region sidecars: **us-east-1 (ES 8)**, **eu-west-1 (ES 8)**, +**ap-south-1 (ES 9)**. One Flight SQL endpoint federating three Elasticsearch +clusters across regions and ES major versions. + +## ⚠️ Licensing — a Pro (or Enterprise) license is REQUIRED + +3 sidecars = 3 clusters. The federation enforces a per-platform cluster **quota** +at startup (it is the `maxClusters` quota, **not** a feature flag): + +| Clusters (sidecars) | Community (no license) | Pro | Enterprise | +|---|---|---|---| +| 1 | ✅ Ready (maxClusters=1) | ✅ | ✅ | +| 2–5 | ❌ federation CrashLoops (sys.exit) | ✅ (maxClusters=5) | ✅ | +| 6+ | ❌ | ❌ | ✅ (unlimited) | + +So this example **will not boot** without a Pro/Enterprise license. Create the +license Secret BEFORE `helm install`: + +```bash +kubectl create secret generic sc4es-pro-license --from-literal=license-key="$SC4ES_PRO_JWT" +``` + +(The federation surfaces the upgrade URL it emits at startup if you forget.) + +> ⚠️ **A Pro JWT alone is not enough — the federation IMAGE must be able to verify it.** +> Offline JWT verification is performed by a closed-source license manager that is +> bundled ONLY in the Pro/Enterprise-capable federation image. The Community/OSS image +> ships only the Community license manager, so it ignores any injected JWT, resolves to +> Community (`maxClusters=1`), and **still CrashLoops** on this 3-cluster example. The +> image must ALSO be able to resolve the license SIGNING public key offline (env +> `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY`, or the issuer's JWKS endpoint must be reachable). +> Use a Pro/Enterprise-entitled federation image + provision the public key; consult the +> licensing/operator guide for the exact image + key provisioning. This precondition is +> tracked for the CI/install path (Story 16.5 FACT F / Story 16.1 OQ-5). A bare +> `--set image.tag` of the OSS snapshot image WILL CrashLoop even with a valid JWT Secret. + +## When to use this — three wedges + +- **GDPR data residency.** `eu-west-1` keeps EU data in the EU; the federation can + still run a cross-cluster JOIN that *references* it without copying rows across + borders (the JOIN executes in the federation; each region's data stays home). +- **SRE incident triage** (R1 marketing wedge). During an incident, one Flight SQL + endpoint lets an SRE `JOIN` logs/metrics/traces indices that live in **different + regional clusters** — no per-cluster console hopping, no ETL. A single query + correlates `us-east-1.errors` with `ap-south-1.upstream_latency`. +- **Mixed-version migration window.** Run ES 8 in `us-east-1` + `eu-west-1` while + `ap-south-1` is migrated to ES 9 — the federation routes across versions + transparently (the per-region sidecar image is chosen from `elasticsearchVersion`). + No "big bang" cutover; migrate one region at a time and keep federating throughout. + +## Topology + +``` + ┌──────────────────────────────────┐ + Flight SQL client │ Federation (Flight SQL :32020) │ + (BI / ADBC / JDBC) ──┼─► softclient4es-federation │ + │ health gRPC :32021 (aggregate) │ + └───┬────────────┬────────────┬─────┘ + │ │ │ in-cluster gRPC + ┌─────────────▼──┐ ┌──────▼───────┐ ┌─▼──────────────┐ + │ us-east-1 │ │ eu-west-1 │ │ ap-south-1 │ + │ sidecar (ES 8) │ │ sidecar(ES 8)│ │ sidecar (ES 9) │ + │ :32010 default │ │ :32010 │ │ :32010 │ + └────────┬───────┘ └──────┬───────┘ └───────┬────────┘ + ▼ ▼ ▼ + ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ + │ ES 8 us-east-1│ │ ES 8 eu-west-1│ │ ES 9 ap-south-1│ + │ (US data) │ │ (EU data/GDPR) │ │ (migrating→ES9)│ + └────────────────┘ └────────────────┘ └────────────────┘ +``` + +## Install + +```bash +# 1. Pro license Secret (REQUIRED): +kubectl create secret generic sc4es-pro-license --from-literal=license-key="$SC4ES_PRO_JWT" + +# 2. Per-region ES + sidecar-auth Secrets (chart does NOT create these): +for r in us-east-1 eu-west-1 ap-south-1; do + kubectl create secret generic sc4es-es-$r \ + --from-literal=es-auth-method=basic --from-literal=es-username=elastic --from-literal=es-password='' + kubectl create secret generic sc4es-arrow-$r --from-literal=arrow-bearer-token='' +done + +# 3. Install: +helm install softclient4es-federation softclient4es-federation \ + -f examples/three-region/values.yaml +``` + +## Verify + +```bash +helm test softclient4es-federation # SHOW CATALOGS returns 3 +``` + +## Readiness & the all-or-nothing aggregate (important for multi-region) + +With `federation.probes.useGrpc: true` (default, K8s ≥ 1.27) the federation's gRPC +readiness aggregate is **all-or-nothing**: if **ANY ONE** region's sidecar is +unreachable, the federation Pod goes **NotReady** and is pulled from its Service — +so **every** query fails, including ones targeting the healthy regions. This is +"fail-closed" routing. For multi-region production where partial availability is +preferable (keep serving the healthy regions, fail only the queries that touch the +down region), set `federation.probes.useGrpc: false` (TCP readiness — the federation +stays Ready and degrades per-query). On K8s < 1.27 you MUST use `useGrpc: false`. diff --git a/softclient4es-federation/examples/three-region/values.yaml b/softclient4es-federation/examples/three-region/values.yaml new file mode 100644 index 0000000..aa61d8d --- /dev/null +++ b/softclient4es-federation/examples/three-region/values.yaml @@ -0,0 +1,63 @@ +# ───────────────────────────────────────────────────────────────────────────── +# SoftClient4ES Federation — THREE-REGION (mixed ES versions) example. +# +# Federation + 3 sidecars across us-east-1 (ES 8), eu-west-1 (ES 8), ap-south-1 (ES 9). +# Demonstrates GDPR data residency, cross-cluster JOIN, SRE incident triage, and a +# mixed-ES-version migration window (two regions on ES 8 while ap-south-1 is on ES 9). +# +# ⚠️ REQUIRES A PRO (or Enterprise) LICENSE. +# 3 sidecars = 3 clusters; Community caps at maxClusters=1, so WITHOUT a Pro/ +# Enterprise license the federation Pod CrashLoops by design (startup quota guard). +# Create a Secret `sc4es-pro-license` (data key `license-key` = your JWT) first. +# +# Install: +# helm install softclient4es-federation softclient4es-federation \ +# -f examples/three-region/values.yaml +# ───────────────────────────────────────────────────────────────────────────── + +# Pro/Enterprise license — REQUIRED for >=2 sidecars (maxClusters: Community=1, Pro=5, Ent=∞). +license: + secretName: "sc4es-pro-license" # data key `license-key` = JWT (override via licenseKeyKey) + +telemetry: + enabled: true + +federation: + probes: + # All-or-nothing gRPC readiness aggregate (VERIFIED): ANY unreachable sidecar + # makes the federation NotReady → ALL queries fail (incl. healthy regions). + # For multi-region partial-availability, set useGrpc: false (TCP readiness) — + # see README "Readiness & the all-or-nothing aggregate". + useGrpc: true + +sidecars: + # ── us-east-1 (Elasticsearch 8) — default region for bare-table routing ── + - name: us-east-1 + elasticsearchVersion: 8 + elasticsearch: + url: "https://es-us-east-1.example.com:9200" + credentialsSecretName: "sc4es-es-us-east-1" + auth: + method: bearer # sidecar incoming auth (single source of truth) + credentialsSecretName: "sc4es-arrow-us-east-1" # arrow-bearer-token -> ARROW_AUTH_* + CONFIG_FORCE_* + default: true + + # ── eu-west-1 (Elasticsearch 8) — GDPR data-residency region ── + - name: eu-west-1 + elasticsearchVersion: 8 + elasticsearch: + url: "https://es-eu-west-1.example.com:9200" + credentialsSecretName: "sc4es-es-eu-west-1" + auth: + method: bearer + credentialsSecretName: "sc4es-arrow-eu-west-1" + + # ── ap-south-1 (Elasticsearch 9) — region mid-migration to ES 9 ── + - name: ap-south-1 + elasticsearchVersion: 9 + elasticsearch: + url: "https://es-ap-south-1.example.com:9200" + credentialsSecretName: "sc4es-es-ap-south-1" + auth: + method: bearer + credentialsSecretName: "sc4es-arrow-ap-south-1" diff --git a/softclient4es-federation/tests/golden/example-heterogeneous-ready.yaml b/softclient4es-federation/tests/golden/example-heterogeneous-ready.yaml new file mode 100644 index 0000000..e305683 --- /dev/null +++ b/softclient4es-federation/tests/golden/example-heterogeneous-ready.yaml @@ -0,0 +1,948 @@ +--- +# Source: softclient4es-federation/templates/serviceaccount.yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: fed-softclient4es-federation + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es +automountServiceAccountToken: true +--- +# Source: softclient4es-federation/templates/federation-configmap.yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: fed-softclient4es-federation-config + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es +data: + application.conf: | + # GENERATED by the softclient4es-federation Helm chart — do not edit in-cluster. + # Populates arrow.flight.federation.servers from values.yaml `sidecars[]`. + # The scalar FEDERATION_* settings are still supplied via env vars on the + # Deployment; this file ONLY carries the `servers` map (which has no env override). + arrow.flight.federation { + servers { + us-east-1 { + host = "fed-softclient4es-federation-us-east-1.default.svc.cluster.local" + port = 32010 + default = true + # Federation OUTGOING auth to this sidecar — single source of truth with the + # sidecar's ARROW_AUTH_* env. For a Secret-backed credential (credentialsSecretName) + # the VALUE is absent here and arrives via CONFIG_FORCE_* (override_with_env_vars) on + # the federation Deployment (Story 16.3, FACT A) — this block then renders method-only, + # and the override forces the value before FederationConfig.validate() runs. Inline + # values (dev/test) are rendered when present. + credentials { + method = "bearer" + } + } + eu-west-1 { + host = "fed-softclient4es-federation-eu-west-1.default.svc.cluster.local" + port = 32010 + # Federation OUTGOING auth to this sidecar — single source of truth with the + # sidecar's ARROW_AUTH_* env. For a Secret-backed credential (credentialsSecretName) + # the VALUE is absent here and arrives via CONFIG_FORCE_* (override_with_env_vars) on + # the federation Deployment (Story 16.3, FACT A) — this block then renders method-only, + # and the override forces the value before FederationConfig.validate() runs. Inline + # values (dev/test) are rendered when present. + credentials { + method = "bearer" + } + } + ap-south-1 { + host = "fed-softclient4es-federation-ap-south-1.default.svc.cluster.local" + port = 32010 + # Federation OUTGOING auth to this sidecar — single source of truth with the + # sidecar's ARROW_AUTH_* env. For a Secret-backed credential (credentialsSecretName) + # the VALUE is absent here and arrives via CONFIG_FORCE_* (override_with_env_vars) on + # the federation Deployment (Story 16.3, FACT A) — this block then renders method-only, + # and the override forces the value before FederationConfig.validate() runs. Inline + # values (dev/test) are rendered when present. + credentials { + method = "bearer" + } + } + } + } +--- +# Source: softclient4es-federation/templates/service.yaml +apiVersion: v1 +kind: Service +metadata: + name: fed-softclient4es-federation + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es +spec: + type: ClusterIP + ports: + - name: flight-sql + port: 32020 + targetPort: flight-sql + protocol: TCP + selector: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed +--- +# Source: softclient4es-federation/templates/sidecar-service.yaml +apiVersion: v1 +kind: Service +metadata: + name: fed-softclient4es-federation-us-east-1 + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: us-east-1 + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + type: ClusterIP + ports: + - name: flight-sql + port: 32010 + targetPort: flight-sql + protocol: TCP + selector: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: us-east-1 +--- +# Source: softclient4es-federation/templates/sidecar-service.yaml +apiVersion: v1 +kind: Service +metadata: + name: fed-softclient4es-federation-eu-west-1 + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: eu-west-1 + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + type: ClusterIP + ports: + - name: flight-sql + port: 32010 + targetPort: flight-sql + protocol: TCP + selector: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: eu-west-1 +--- +# Source: softclient4es-federation/templates/sidecar-service.yaml +apiVersion: v1 +kind: Service +metadata: + name: fed-softclient4es-federation-ap-south-1 + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: ap-south-1 + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + type: ClusterIP + ports: + - name: flight-sql + port: 32010 + targetPort: flight-sql + protocol: TCP + selector: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: ap-south-1 +--- +# Source: softclient4es-federation/templates/deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: fed-softclient4es-federation + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + template: + metadata: + labels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + annotations: + # Roll the federation Pod whenever the rendered servers ConfigMap changes + # (add/remove/edit a sidecar) so `helm upgrade` actually takes effect. + checksum/config: e8fc04a9ca01bd0f0ff08964beca0bee9a2da40b729f47d4aa0b57a3fb2bac95 + spec: + serviceAccountName: fed-softclient4es-federation + securityContext: + fsGroup: 1001 + runAsGroup: 1001 + runAsNonRoot: true + runAsUser: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: federation + image: "docker.io/softnetwork/softclient4es-federation:0.2.0" + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1001 + ports: + - name: flight-sql + containerPort: 32020 + protocol: TCP + - name: health + containerPort: 32021 + protocol: TCP + env: + - name: FEDERATION_HOST + value: "0.0.0.0" + - name: FEDERATION_PORT + value: "32020" + - name: FEDERATION_MAX_MEMORY + value: "512m" + - name: FEDERATION_QUERY_TIMEOUT + value: "30" + - name: FEDERATION_HEALTH_PORT + value: "32021" + - name: FEDERATION_HEALTH_PROBE_TIMEOUT + value: "5" + - name: FEDERATION_DUCKDB_PATH + value: ":memory:" + - name: FEDERATION_UPGRADE_URL + value: "https://portal.softclient4es.com/pricing" + - name: SOFTCLIENT4ES_TELEMETRY_ENABLED + value: "true" + - name: SOFTCLIENT4ES_LICENSE_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: license-key + optional: true + - name: SOFTCLIENT4ES_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: api-key + optional: true + # Point Typesafe Config at the mounted ConfigMap so ConfigFactory.load() + # picks up the `servers` map (which has NO env override). The native-packager + # bash launcher reads $JAVA_OPTS at runtime (VERIFIED) and prepends it to the + # JVM args; reference.conf still loads, so the FEDERATION_* env defaults stay. + # Story 16.3: when any sidecar uses Secret-backed auth, also enable + # override_with_env_vars so the CONFIG_FORCE_* env below force the credential + # leaf onto the loaded config BEFORE FederationConfig.validate() runs. + - name: JAVA_OPTS + value: "-Dconfig.file=/opt/docker/conf/application.conf -Dconfig.override_with_env_vars=true" + - name: CONFIG_FORCE_arrow_flight_federation_servers_us__east__1_credentials_bearer__token + valueFrom: + secretKeyRef: + name: sc4es-arrow-us-east-1 + key: arrow-bearer-token + optional: true + - name: CONFIG_FORCE_arrow_flight_federation_servers_eu__west__1_credentials_bearer__token + valueFrom: + secretKeyRef: + name: sc4es-arrow-eu-west-1 + key: arrow-bearer-token + optional: true + - name: CONFIG_FORCE_arrow_flight_federation_servers_ap__south__1_credentials_bearer__token + valueFrom: + secretKeyRef: + name: sc4es-arrow-ap-south-1 + key: arrow-bearer-token + optional: true + # --- Probes (FACT A / A1): TCP-socket only for the skeleton. --- + # The federation health endpoint is gRPC grpc.health.v1.Health and returns + # NOT_SERVING while servers={} (HealthService.scala:105-111), so a gRPC Check + # probe would fail. A TCP probe proves the process is listening, which is the + # correct liveness/readiness signal for a no-downstream skeleton. Story 16.2 + # switches readiness to a gRPC probe once sidecars exist. + livenessProbe: + tcpSocket: + port: health + initialDelaySeconds: 20 + periodSeconds: 15 + timeoutSeconds: 5 + failureThreshold: 3 + # Readiness flips to a native gRPC probe once sidecars make the SERVING + # aggregate meaningful (Story 16.2). ⚠️ ALL-OR-NOTHING: any one unreachable + # downstream → NotReady → ALL queries fail (fail-closed). Set + # federation.probes.useGrpc=false to keep TCP readiness (partial availability; + # required on K8s < 1.27 — native gRPC probes are GA only from 1.27). + readinessProbe: + grpc: + port: 32021 + initialDelaySeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + resources: + limits: + cpu: 1000m + memory: 2Gi + requests: + cpu: 500m + memory: 1Gi + volumeMounts: + - name: scratch + mountPath: /tmp + - name: federation-config + mountPath: /opt/docker/conf + readOnly: true + volumes: + # readOnlyRootFilesystem=true requires a writable scratch dir: the org.duckdb + # JDBC driver extracts its native libduckdb_java library to java.io.tmpdir (/tmp) + # on every boot (PersistentDuckDB is instantiated unconditionally — even with + # path=":memory:"), and the federation stages CTAS / INSERT-with-JOIN Parquet + # under java.io.tmpdir (FederationConfig.scala:266-267 → /tmp/softclient4es/scratch). + # A file DuckDB path (federation.duckdb.path != ":memory:") would also live here. + - name: scratch + emptyDir: + sizeLimit: 2Gi + - name: federation-config + configMap: + name: fed-softclient4es-federation-config +--- +# Source: softclient4es-federation/templates/sidecar-deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: fed-softclient4es-federation-us-east-1 + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: us-east-1 + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: us-east-1 + template: + metadata: + labels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: us-east-1 + spec: + serviceAccountName: fed-softclient4es-federation + securityContext: + fsGroup: 1001 + runAsGroup: 1001 + runAsNonRoot: true + runAsUser: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: sidecar + image: "docker.io/softnetwork/softclient4es8-arrow-flight-sql:0.2.0" + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1001 + ports: + - name: flight-sql + containerPort: 32010 + protocol: TCP + env: + # ── Arrow Flight SQL server (arrow.flight.*) ── + - name: ARROW_HOST + value: "0.0.0.0" + - name: ARROW_PORT + value: "32010" + - name: ARROW_BATCH_SIZE + value: "1000" + - name: ARROW_QUERY_TIMEOUT_SECONDS + value: "120" + - name: ARROW_JOIN_MAX_MEMORY + value: "256m" + - name: ARROW_JOIN_UPGRADE_URL + value: "https://portal.softclient4es.com/pricing" + # ── Elasticsearch connection (elastic.credentials.* — VERIFIED, NOT a single URL) ── + - name: ELASTIC_SCHEME + value: "https" + - name: ELASTIC_HOST + value: "es-us-east-1.example.com" + - name: ELASTIC_PORT + value: "9200" + - name: ELASTIC_AUTH_METHOD + valueFrom: + secretKeyRef: + name: sc4es-es-us-east-1 + key: es-auth-method + optional: true + - name: ELASTIC_CREDENTIALS_USERNAME + valueFrom: + secretKeyRef: + name: sc4es-es-us-east-1 + key: es-username + optional: true + - name: ELASTIC_CREDENTIALS_PASSWORD + valueFrom: + secretKeyRef: + name: sc4es-es-us-east-1 + key: es-password + optional: true + - name: ELASTIC_CREDENTIALS_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-es-us-east-1 + key: es-api-key + optional: true + - name: ELASTIC_CREDENTIALS_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sc4es-es-us-east-1 + key: es-bearer-token + optional: true + # ── Sidecar SERVER-SIDE auth (incoming) — single source of truth with the + # federation's outgoing servers..credentials (rendered in the ConfigMap) ── + - name: ARROW_AUTH_METHOD + value: "bearer" + - name: ARROW_AUTH_USERNAME + valueFrom: + secretKeyRef: + name: sc4es-arrow-us-east-1 + key: arrow-username + optional: true + - name: ARROW_AUTH_PASSWORD + valueFrom: + secretKeyRef: + name: sc4es-arrow-us-east-1 + key: arrow-password + optional: true + - name: ARROW_AUTH_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sc4es-arrow-us-east-1 + key: arrow-bearer-token + optional: true + - name: ARROW_AUTH_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-arrow-us-east-1 + key: arrow-api-key + optional: true + # ── Telemetry opt-out (shared with federation) ── + # VERIFIED: SOFTCLIENT4ES_TELEMETRY_ENABLED is honored by the sidecar too — the env + # override lives in elasticsql licensing reference.conf:30 (softclient4es.telemetry.enabled), + # read by TelemetryConfig.load, on which the arrow sidecar depends transitively. NOTE it is + # NOT in arrowServerSettings.dockerEnvVars (build.sbt:243-255 bakes only ELASTIC_*/ARROW_*), + # so the chart MUST set it explicitly here for the opt-out to reach the sidecar. + - name: SOFTCLIENT4ES_TELEMETRY_ENABLED + value: "true" + - name: SOFTCLIENT4ES_LICENSE_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: license-key + optional: true + - name: SOFTCLIENT4ES_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: api-key + optional: true + # TCP probes — the sidecar has NO gRPC health service (VERIFIED); a raw TCP + # connect to ARROW_PORT proves the Flight SQL server is listening. + livenessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 20 + periodSeconds: 15 + failureThreshold: 3 + readinessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + resources: + limits: + cpu: 1000m + memory: 1536Mi + requests: + cpu: 500m + memory: 768Mi + volumeMounts: + - name: scratch + mountPath: /tmp + volumes: + # Writable /tmp: the sidecar runs the same-cluster JOIN engine on DuckDB + # (DuckDBJoinExecutor via LocalJoinHandler — VERIFIED), whose org.duckdb JDBC + # driver extracts its native libduckdb_java to java.io.tmpdir on first JOIN. + # readOnlyRootFilesystem=true without this would crash the first JOIN query. + - name: scratch + emptyDir: + sizeLimit: 1Gi +--- +# Source: softclient4es-federation/templates/sidecar-deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: fed-softclient4es-federation-eu-west-1 + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: eu-west-1 + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: eu-west-1 + template: + metadata: + labels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: eu-west-1 + spec: + serviceAccountName: fed-softclient4es-federation + securityContext: + fsGroup: 1001 + runAsGroup: 1001 + runAsNonRoot: true + runAsUser: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: sidecar + image: "docker.io/softnetwork/softclient4es8-arrow-flight-sql:0.2.0" + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1001 + ports: + - name: flight-sql + containerPort: 32010 + protocol: TCP + env: + # ── Arrow Flight SQL server (arrow.flight.*) ── + - name: ARROW_HOST + value: "0.0.0.0" + - name: ARROW_PORT + value: "32010" + - name: ARROW_BATCH_SIZE + value: "1000" + - name: ARROW_QUERY_TIMEOUT_SECONDS + value: "120" + - name: ARROW_JOIN_MAX_MEMORY + value: "256m" + - name: ARROW_JOIN_UPGRADE_URL + value: "https://portal.softclient4es.com/pricing" + # ── Elasticsearch connection (elastic.credentials.* — VERIFIED, NOT a single URL) ── + - name: ELASTIC_SCHEME + value: "https" + - name: ELASTIC_HOST + value: "es-eu-west-1.example.com" + - name: ELASTIC_PORT + value: "9200" + - name: ELASTIC_AUTH_METHOD + valueFrom: + secretKeyRef: + name: sc4es-es-eu-west-1 + key: es-auth-method + optional: true + - name: ELASTIC_CREDENTIALS_USERNAME + valueFrom: + secretKeyRef: + name: sc4es-es-eu-west-1 + key: es-username + optional: true + - name: ELASTIC_CREDENTIALS_PASSWORD + valueFrom: + secretKeyRef: + name: sc4es-es-eu-west-1 + key: es-password + optional: true + - name: ELASTIC_CREDENTIALS_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-es-eu-west-1 + key: es-api-key + optional: true + - name: ELASTIC_CREDENTIALS_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sc4es-es-eu-west-1 + key: es-bearer-token + optional: true + # ── Sidecar SERVER-SIDE auth (incoming) — single source of truth with the + # federation's outgoing servers..credentials (rendered in the ConfigMap) ── + - name: ARROW_AUTH_METHOD + value: "bearer" + - name: ARROW_AUTH_USERNAME + valueFrom: + secretKeyRef: + name: sc4es-arrow-eu-west-1 + key: arrow-username + optional: true + - name: ARROW_AUTH_PASSWORD + valueFrom: + secretKeyRef: + name: sc4es-arrow-eu-west-1 + key: arrow-password + optional: true + - name: ARROW_AUTH_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sc4es-arrow-eu-west-1 + key: arrow-bearer-token + optional: true + - name: ARROW_AUTH_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-arrow-eu-west-1 + key: arrow-api-key + optional: true + # ── Telemetry opt-out (shared with federation) ── + # VERIFIED: SOFTCLIENT4ES_TELEMETRY_ENABLED is honored by the sidecar too — the env + # override lives in elasticsql licensing reference.conf:30 (softclient4es.telemetry.enabled), + # read by TelemetryConfig.load, on which the arrow sidecar depends transitively. NOTE it is + # NOT in arrowServerSettings.dockerEnvVars (build.sbt:243-255 bakes only ELASTIC_*/ARROW_*), + # so the chart MUST set it explicitly here for the opt-out to reach the sidecar. + - name: SOFTCLIENT4ES_TELEMETRY_ENABLED + value: "true" + - name: SOFTCLIENT4ES_LICENSE_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: license-key + optional: true + - name: SOFTCLIENT4ES_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: api-key + optional: true + # TCP probes — the sidecar has NO gRPC health service (VERIFIED); a raw TCP + # connect to ARROW_PORT proves the Flight SQL server is listening. + livenessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 20 + periodSeconds: 15 + failureThreshold: 3 + readinessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + resources: + limits: + cpu: 1000m + memory: 1536Mi + requests: + cpu: 500m + memory: 768Mi + volumeMounts: + - name: scratch + mountPath: /tmp + volumes: + # Writable /tmp: the sidecar runs the same-cluster JOIN engine on DuckDB + # (DuckDBJoinExecutor via LocalJoinHandler — VERIFIED), whose org.duckdb JDBC + # driver extracts its native libduckdb_java to java.io.tmpdir on first JOIN. + # readOnlyRootFilesystem=true without this would crash the first JOIN query. + - name: scratch + emptyDir: + sizeLimit: 1Gi +--- +# Source: softclient4es-federation/templates/sidecar-deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: fed-softclient4es-federation-ap-south-1 + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: ap-south-1 + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: ap-south-1 + template: + metadata: + labels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: ap-south-1 + spec: + serviceAccountName: fed-softclient4es-federation + securityContext: + fsGroup: 1001 + runAsGroup: 1001 + runAsNonRoot: true + runAsUser: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: sidecar + image: "docker.io/softnetwork/softclient4es9-arrow-flight-sql:0.2.0" + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1001 + ports: + - name: flight-sql + containerPort: 32010 + protocol: TCP + env: + # ── Arrow Flight SQL server (arrow.flight.*) ── + - name: ARROW_HOST + value: "0.0.0.0" + - name: ARROW_PORT + value: "32010" + - name: ARROW_BATCH_SIZE + value: "1000" + - name: ARROW_QUERY_TIMEOUT_SECONDS + value: "120" + - name: ARROW_JOIN_MAX_MEMORY + value: "256m" + - name: ARROW_JOIN_UPGRADE_URL + value: "https://portal.softclient4es.com/pricing" + # ── Elasticsearch connection (elastic.credentials.* — VERIFIED, NOT a single URL) ── + - name: ELASTIC_SCHEME + value: "https" + - name: ELASTIC_HOST + value: "es-ap-south-1.example.com" + - name: ELASTIC_PORT + value: "9200" + - name: ELASTIC_AUTH_METHOD + valueFrom: + secretKeyRef: + name: sc4es-es-ap-south-1 + key: es-auth-method + optional: true + - name: ELASTIC_CREDENTIALS_USERNAME + valueFrom: + secretKeyRef: + name: sc4es-es-ap-south-1 + key: es-username + optional: true + - name: ELASTIC_CREDENTIALS_PASSWORD + valueFrom: + secretKeyRef: + name: sc4es-es-ap-south-1 + key: es-password + optional: true + - name: ELASTIC_CREDENTIALS_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-es-ap-south-1 + key: es-api-key + optional: true + - name: ELASTIC_CREDENTIALS_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sc4es-es-ap-south-1 + key: es-bearer-token + optional: true + # ── Sidecar SERVER-SIDE auth (incoming) — single source of truth with the + # federation's outgoing servers..credentials (rendered in the ConfigMap) ── + - name: ARROW_AUTH_METHOD + value: "bearer" + - name: ARROW_AUTH_USERNAME + valueFrom: + secretKeyRef: + name: sc4es-arrow-ap-south-1 + key: arrow-username + optional: true + - name: ARROW_AUTH_PASSWORD + valueFrom: + secretKeyRef: + name: sc4es-arrow-ap-south-1 + key: arrow-password + optional: true + - name: ARROW_AUTH_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sc4es-arrow-ap-south-1 + key: arrow-bearer-token + optional: true + - name: ARROW_AUTH_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-arrow-ap-south-1 + key: arrow-api-key + optional: true + # ── Telemetry opt-out (shared with federation) ── + # VERIFIED: SOFTCLIENT4ES_TELEMETRY_ENABLED is honored by the sidecar too — the env + # override lives in elasticsql licensing reference.conf:30 (softclient4es.telemetry.enabled), + # read by TelemetryConfig.load, on which the arrow sidecar depends transitively. NOTE it is + # NOT in arrowServerSettings.dockerEnvVars (build.sbt:243-255 bakes only ELASTIC_*/ARROW_*), + # so the chart MUST set it explicitly here for the opt-out to reach the sidecar. + - name: SOFTCLIENT4ES_TELEMETRY_ENABLED + value: "true" + - name: SOFTCLIENT4ES_LICENSE_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: license-key + optional: true + - name: SOFTCLIENT4ES_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: api-key + optional: true + # TCP probes — the sidecar has NO gRPC health service (VERIFIED); a raw TCP + # connect to ARROW_PORT proves the Flight SQL server is listening. + livenessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 20 + periodSeconds: 15 + failureThreshold: 3 + readinessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + resources: + limits: + cpu: 1000m + memory: 1536Mi + requests: + cpu: 500m + memory: 768Mi + volumeMounts: + - name: scratch + mountPath: /tmp + volumes: + # Writable /tmp: the sidecar runs the same-cluster JOIN engine on DuckDB + # (DuckDBJoinExecutor via LocalJoinHandler — VERIFIED), whose org.duckdb JDBC + # driver extracts its native libduckdb_java to java.io.tmpdir on first JOIN. + # readOnlyRootFilesystem=true without this would crash the first JOIN query. + - name: scratch + emptyDir: + sizeLimit: 1Gi +--- +# Source: softclient4es-federation/templates/tests/smoke-test.yaml +apiVersion: v1 +kind: Pod +metadata: + name: fed-softclient4es-federation-test-catalogs + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es + annotations: + "helm.sh/hook": test + "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded +spec: + restartPolicy: Never + containers: + - name: show-catalogs + image: "python:3.12-slim" + command: ["sh", "-c"] + args: + - | + pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + python - <<'PY' + import sys + import adbc_driver_flightsql.dbapi as dbapi + host = "fed-softclient4es-federation" + conn = dbapi.connect(f"grpc://{host}:32020") + # GetCatalogs: one catalog per registered sidecar alias. + cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + expected = 3 + print("catalogs:", cats) + if len(cats) != expected: + print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) + sys.exit(1) + print("OK") + PY diff --git a/softclient4es-federation/tests/golden/example-single-cluster.yaml b/softclient4es-federation/tests/golden/example-single-cluster.yaml new file mode 100644 index 0000000..dbafbab --- /dev/null +++ b/softclient4es-federation/tests/golden/example-single-cluster.yaml @@ -0,0 +1,421 @@ +--- +# Source: softclient4es-federation/templates/serviceaccount.yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: fed-softclient4es-federation + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es +automountServiceAccountToken: true +--- +# Source: softclient4es-federation/templates/federation-configmap.yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: fed-softclient4es-federation-config + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es +data: + application.conf: | + # GENERATED by the softclient4es-federation Helm chart — do not edit in-cluster. + # Populates arrow.flight.federation.servers from values.yaml `sidecars[]`. + # The scalar FEDERATION_* settings are still supplied via env vars on the + # Deployment; this file ONLY carries the `servers` map (which has no env override). + arrow.flight.federation { + servers { + primary { + host = "fed-softclient4es-federation-primary.default.svc.cluster.local" + port = 32010 + default = true + } + } + } +--- +# Source: softclient4es-federation/templates/service.yaml +apiVersion: v1 +kind: Service +metadata: + name: fed-softclient4es-federation + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es +spec: + type: ClusterIP + ports: + - name: flight-sql + port: 32020 + targetPort: flight-sql + protocol: TCP + selector: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed +--- +# Source: softclient4es-federation/templates/sidecar-service.yaml +apiVersion: v1 +kind: Service +metadata: + name: fed-softclient4es-federation-primary + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: primary + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + type: ClusterIP + ports: + - name: flight-sql + port: 32010 + targetPort: flight-sql + protocol: TCP + selector: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: primary +--- +# Source: softclient4es-federation/templates/deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: fed-softclient4es-federation + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + template: + metadata: + labels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + annotations: + # Roll the federation Pod whenever the rendered servers ConfigMap changes + # (add/remove/edit a sidecar) so `helm upgrade` actually takes effect. + checksum/config: 1b75346d97567cae377a26f4e7f2e8d5c6920da45e8e7f9ca4e4b986169ece2b + spec: + serviceAccountName: fed-softclient4es-federation + securityContext: + fsGroup: 1001 + runAsGroup: 1001 + runAsNonRoot: true + runAsUser: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: federation + image: "docker.io/softnetwork/softclient4es-federation:0.2.0" + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1001 + ports: + - name: flight-sql + containerPort: 32020 + protocol: TCP + - name: health + containerPort: 32021 + protocol: TCP + env: + - name: FEDERATION_HOST + value: "0.0.0.0" + - name: FEDERATION_PORT + value: "32020" + - name: FEDERATION_MAX_MEMORY + value: "512m" + - name: FEDERATION_QUERY_TIMEOUT + value: "30" + - name: FEDERATION_HEALTH_PORT + value: "32021" + - name: FEDERATION_HEALTH_PROBE_TIMEOUT + value: "5" + - name: FEDERATION_DUCKDB_PATH + value: ":memory:" + - name: FEDERATION_UPGRADE_URL + value: "https://portal.softclient4es.com/pricing" + - name: SOFTCLIENT4ES_TELEMETRY_ENABLED + value: "true" + # Point Typesafe Config at the mounted ConfigMap so ConfigFactory.load() + # picks up the `servers` map (which has NO env override). The native-packager + # bash launcher reads $JAVA_OPTS at runtime (VERIFIED) and prepends it to the + # JVM args; reference.conf still loads, so the FEDERATION_* env defaults stay. + # Story 16.3: when any sidecar uses Secret-backed auth, also enable + # override_with_env_vars so the CONFIG_FORCE_* env below force the credential + # leaf onto the loaded config BEFORE FederationConfig.validate() runs. + - name: JAVA_OPTS + value: "-Dconfig.file=/opt/docker/conf/application.conf" + # --- Probes (FACT A / A1): TCP-socket only for the skeleton. --- + # The federation health endpoint is gRPC grpc.health.v1.Health and returns + # NOT_SERVING while servers={} (HealthService.scala:105-111), so a gRPC Check + # probe would fail. A TCP probe proves the process is listening, which is the + # correct liveness/readiness signal for a no-downstream skeleton. Story 16.2 + # switches readiness to a gRPC probe once sidecars exist. + livenessProbe: + tcpSocket: + port: health + initialDelaySeconds: 20 + periodSeconds: 15 + timeoutSeconds: 5 + failureThreshold: 3 + # Readiness flips to a native gRPC probe once sidecars make the SERVING + # aggregate meaningful (Story 16.2). ⚠️ ALL-OR-NOTHING: any one unreachable + # downstream → NotReady → ALL queries fail (fail-closed). Set + # federation.probes.useGrpc=false to keep TCP readiness (partial availability; + # required on K8s < 1.27 — native gRPC probes are GA only from 1.27). + readinessProbe: + grpc: + port: 32021 + initialDelaySeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + resources: + limits: + cpu: 1000m + memory: 2Gi + requests: + cpu: 500m + memory: 1Gi + volumeMounts: + - name: scratch + mountPath: /tmp + - name: federation-config + mountPath: /opt/docker/conf + readOnly: true + volumes: + # readOnlyRootFilesystem=true requires a writable scratch dir: the org.duckdb + # JDBC driver extracts its native libduckdb_java library to java.io.tmpdir (/tmp) + # on every boot (PersistentDuckDB is instantiated unconditionally — even with + # path=":memory:"), and the federation stages CTAS / INSERT-with-JOIN Parquet + # under java.io.tmpdir (FederationConfig.scala:266-267 → /tmp/softclient4es/scratch). + # A file DuckDB path (federation.duckdb.path != ":memory:") would also live here. + - name: scratch + emptyDir: + sizeLimit: 2Gi + - name: federation-config + configMap: + name: fed-softclient4es-federation-config +--- +# Source: softclient4es-federation/templates/sidecar-deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: fed-softclient4es-federation-primary + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: primary + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: primary + template: + metadata: + labels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: primary + spec: + serviceAccountName: fed-softclient4es-federation + securityContext: + fsGroup: 1001 + runAsGroup: 1001 + runAsNonRoot: true + runAsUser: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: sidecar + image: "docker.io/softnetwork/softclient4es8-arrow-flight-sql:0.2.0" + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1001 + ports: + - name: flight-sql + containerPort: 32010 + protocol: TCP + env: + # ── Arrow Flight SQL server (arrow.flight.*) ── + - name: ARROW_HOST + value: "0.0.0.0" + - name: ARROW_PORT + value: "32010" + - name: ARROW_BATCH_SIZE + value: "1000" + - name: ARROW_QUERY_TIMEOUT_SECONDS + value: "120" + - name: ARROW_JOIN_MAX_MEMORY + value: "256m" + - name: ARROW_JOIN_UPGRADE_URL + value: "https://portal.softclient4es.com/pricing" + # ── Elasticsearch connection (elastic.credentials.* — VERIFIED, NOT a single URL) ── + - name: ELASTIC_SCHEME + value: "https" + - name: ELASTIC_HOST + value: "es.example.com" + - name: ELASTIC_PORT + value: "9200" + - name: ELASTIC_AUTH_METHOD + valueFrom: + secretKeyRef: + name: sc4es-es-credentials + key: es-auth-method + optional: true + - name: ELASTIC_CREDENTIALS_USERNAME + valueFrom: + secretKeyRef: + name: sc4es-es-credentials + key: es-username + optional: true + - name: ELASTIC_CREDENTIALS_PASSWORD + valueFrom: + secretKeyRef: + name: sc4es-es-credentials + key: es-password + optional: true + - name: ELASTIC_CREDENTIALS_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-es-credentials + key: es-api-key + optional: true + - name: ELASTIC_CREDENTIALS_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sc4es-es-credentials + key: es-bearer-token + optional: true + # ── Sidecar SERVER-SIDE auth (incoming) — single source of truth with the + # federation's outgoing servers..credentials (rendered in the ConfigMap) ── + - name: ARROW_AUTH_METHOD + value: "none" + # ── Telemetry opt-out (shared with federation) ── + # VERIFIED: SOFTCLIENT4ES_TELEMETRY_ENABLED is honored by the sidecar too — the env + # override lives in elasticsql licensing reference.conf:30 (softclient4es.telemetry.enabled), + # read by TelemetryConfig.load, on which the arrow sidecar depends transitively. NOTE it is + # NOT in arrowServerSettings.dockerEnvVars (build.sbt:243-255 bakes only ELASTIC_*/ARROW_*), + # so the chart MUST set it explicitly here for the opt-out to reach the sidecar. + - name: SOFTCLIENT4ES_TELEMETRY_ENABLED + value: "true" + # TCP probes — the sidecar has NO gRPC health service (VERIFIED); a raw TCP + # connect to ARROW_PORT proves the Flight SQL server is listening. + livenessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 20 + periodSeconds: 15 + failureThreshold: 3 + readinessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + resources: + limits: + cpu: 1000m + memory: 1536Mi + requests: + cpu: 500m + memory: 768Mi + volumeMounts: + - name: scratch + mountPath: /tmp + volumes: + # Writable /tmp: the sidecar runs the same-cluster JOIN engine on DuckDB + # (DuckDBJoinExecutor via LocalJoinHandler — VERIFIED), whose org.duckdb JDBC + # driver extracts its native libduckdb_java to java.io.tmpdir on first JOIN. + # readOnlyRootFilesystem=true without this would crash the first JOIN query. + - name: scratch + emptyDir: + sizeLimit: 1Gi +--- +# Source: softclient4es-federation/templates/tests/smoke-test.yaml +apiVersion: v1 +kind: Pod +metadata: + name: fed-softclient4es-federation-test-catalogs + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es + annotations: + "helm.sh/hook": test + "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded +spec: + restartPolicy: Never + containers: + - name: show-catalogs + image: "python:3.12-slim" + command: ["sh", "-c"] + args: + - | + pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + python - <<'PY' + import sys + import adbc_driver_flightsql.dbapi as dbapi + host = "fed-softclient4es-federation" + conn = dbapi.connect(f"grpc://{host}:32020") + # GetCatalogs: one catalog per registered sidecar alias. + cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + expected = 1 + print("catalogs:", cats) + if len(cats) != expected: + print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) + sys.exit(1) + print("OK") + PY diff --git a/softclient4es-federation/tests/golden/example-three-region.yaml b/softclient4es-federation/tests/golden/example-three-region.yaml new file mode 100644 index 0000000..e305683 --- /dev/null +++ b/softclient4es-federation/tests/golden/example-three-region.yaml @@ -0,0 +1,948 @@ +--- +# Source: softclient4es-federation/templates/serviceaccount.yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: fed-softclient4es-federation + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es +automountServiceAccountToken: true +--- +# Source: softclient4es-federation/templates/federation-configmap.yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: fed-softclient4es-federation-config + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es +data: + application.conf: | + # GENERATED by the softclient4es-federation Helm chart — do not edit in-cluster. + # Populates arrow.flight.federation.servers from values.yaml `sidecars[]`. + # The scalar FEDERATION_* settings are still supplied via env vars on the + # Deployment; this file ONLY carries the `servers` map (which has no env override). + arrow.flight.federation { + servers { + us-east-1 { + host = "fed-softclient4es-federation-us-east-1.default.svc.cluster.local" + port = 32010 + default = true + # Federation OUTGOING auth to this sidecar — single source of truth with the + # sidecar's ARROW_AUTH_* env. For a Secret-backed credential (credentialsSecretName) + # the VALUE is absent here and arrives via CONFIG_FORCE_* (override_with_env_vars) on + # the federation Deployment (Story 16.3, FACT A) — this block then renders method-only, + # and the override forces the value before FederationConfig.validate() runs. Inline + # values (dev/test) are rendered when present. + credentials { + method = "bearer" + } + } + eu-west-1 { + host = "fed-softclient4es-federation-eu-west-1.default.svc.cluster.local" + port = 32010 + # Federation OUTGOING auth to this sidecar — single source of truth with the + # sidecar's ARROW_AUTH_* env. For a Secret-backed credential (credentialsSecretName) + # the VALUE is absent here and arrives via CONFIG_FORCE_* (override_with_env_vars) on + # the federation Deployment (Story 16.3, FACT A) — this block then renders method-only, + # and the override forces the value before FederationConfig.validate() runs. Inline + # values (dev/test) are rendered when present. + credentials { + method = "bearer" + } + } + ap-south-1 { + host = "fed-softclient4es-federation-ap-south-1.default.svc.cluster.local" + port = 32010 + # Federation OUTGOING auth to this sidecar — single source of truth with the + # sidecar's ARROW_AUTH_* env. For a Secret-backed credential (credentialsSecretName) + # the VALUE is absent here and arrives via CONFIG_FORCE_* (override_with_env_vars) on + # the federation Deployment (Story 16.3, FACT A) — this block then renders method-only, + # and the override forces the value before FederationConfig.validate() runs. Inline + # values (dev/test) are rendered when present. + credentials { + method = "bearer" + } + } + } + } +--- +# Source: softclient4es-federation/templates/service.yaml +apiVersion: v1 +kind: Service +metadata: + name: fed-softclient4es-federation + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es +spec: + type: ClusterIP + ports: + - name: flight-sql + port: 32020 + targetPort: flight-sql + protocol: TCP + selector: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed +--- +# Source: softclient4es-federation/templates/sidecar-service.yaml +apiVersion: v1 +kind: Service +metadata: + name: fed-softclient4es-federation-us-east-1 + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: us-east-1 + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + type: ClusterIP + ports: + - name: flight-sql + port: 32010 + targetPort: flight-sql + protocol: TCP + selector: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: us-east-1 +--- +# Source: softclient4es-federation/templates/sidecar-service.yaml +apiVersion: v1 +kind: Service +metadata: + name: fed-softclient4es-federation-eu-west-1 + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: eu-west-1 + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + type: ClusterIP + ports: + - name: flight-sql + port: 32010 + targetPort: flight-sql + protocol: TCP + selector: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: eu-west-1 +--- +# Source: softclient4es-federation/templates/sidecar-service.yaml +apiVersion: v1 +kind: Service +metadata: + name: fed-softclient4es-federation-ap-south-1 + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: ap-south-1 + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + type: ClusterIP + ports: + - name: flight-sql + port: 32010 + targetPort: flight-sql + protocol: TCP + selector: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: ap-south-1 +--- +# Source: softclient4es-federation/templates/deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: fed-softclient4es-federation + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + template: + metadata: + labels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + annotations: + # Roll the federation Pod whenever the rendered servers ConfigMap changes + # (add/remove/edit a sidecar) so `helm upgrade` actually takes effect. + checksum/config: e8fc04a9ca01bd0f0ff08964beca0bee9a2da40b729f47d4aa0b57a3fb2bac95 + spec: + serviceAccountName: fed-softclient4es-federation + securityContext: + fsGroup: 1001 + runAsGroup: 1001 + runAsNonRoot: true + runAsUser: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: federation + image: "docker.io/softnetwork/softclient4es-federation:0.2.0" + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1001 + ports: + - name: flight-sql + containerPort: 32020 + protocol: TCP + - name: health + containerPort: 32021 + protocol: TCP + env: + - name: FEDERATION_HOST + value: "0.0.0.0" + - name: FEDERATION_PORT + value: "32020" + - name: FEDERATION_MAX_MEMORY + value: "512m" + - name: FEDERATION_QUERY_TIMEOUT + value: "30" + - name: FEDERATION_HEALTH_PORT + value: "32021" + - name: FEDERATION_HEALTH_PROBE_TIMEOUT + value: "5" + - name: FEDERATION_DUCKDB_PATH + value: ":memory:" + - name: FEDERATION_UPGRADE_URL + value: "https://portal.softclient4es.com/pricing" + - name: SOFTCLIENT4ES_TELEMETRY_ENABLED + value: "true" + - name: SOFTCLIENT4ES_LICENSE_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: license-key + optional: true + - name: SOFTCLIENT4ES_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: api-key + optional: true + # Point Typesafe Config at the mounted ConfigMap so ConfigFactory.load() + # picks up the `servers` map (which has NO env override). The native-packager + # bash launcher reads $JAVA_OPTS at runtime (VERIFIED) and prepends it to the + # JVM args; reference.conf still loads, so the FEDERATION_* env defaults stay. + # Story 16.3: when any sidecar uses Secret-backed auth, also enable + # override_with_env_vars so the CONFIG_FORCE_* env below force the credential + # leaf onto the loaded config BEFORE FederationConfig.validate() runs. + - name: JAVA_OPTS + value: "-Dconfig.file=/opt/docker/conf/application.conf -Dconfig.override_with_env_vars=true" + - name: CONFIG_FORCE_arrow_flight_federation_servers_us__east__1_credentials_bearer__token + valueFrom: + secretKeyRef: + name: sc4es-arrow-us-east-1 + key: arrow-bearer-token + optional: true + - name: CONFIG_FORCE_arrow_flight_federation_servers_eu__west__1_credentials_bearer__token + valueFrom: + secretKeyRef: + name: sc4es-arrow-eu-west-1 + key: arrow-bearer-token + optional: true + - name: CONFIG_FORCE_arrow_flight_federation_servers_ap__south__1_credentials_bearer__token + valueFrom: + secretKeyRef: + name: sc4es-arrow-ap-south-1 + key: arrow-bearer-token + optional: true + # --- Probes (FACT A / A1): TCP-socket only for the skeleton. --- + # The federation health endpoint is gRPC grpc.health.v1.Health and returns + # NOT_SERVING while servers={} (HealthService.scala:105-111), so a gRPC Check + # probe would fail. A TCP probe proves the process is listening, which is the + # correct liveness/readiness signal for a no-downstream skeleton. Story 16.2 + # switches readiness to a gRPC probe once sidecars exist. + livenessProbe: + tcpSocket: + port: health + initialDelaySeconds: 20 + periodSeconds: 15 + timeoutSeconds: 5 + failureThreshold: 3 + # Readiness flips to a native gRPC probe once sidecars make the SERVING + # aggregate meaningful (Story 16.2). ⚠️ ALL-OR-NOTHING: any one unreachable + # downstream → NotReady → ALL queries fail (fail-closed). Set + # federation.probes.useGrpc=false to keep TCP readiness (partial availability; + # required on K8s < 1.27 — native gRPC probes are GA only from 1.27). + readinessProbe: + grpc: + port: 32021 + initialDelaySeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + resources: + limits: + cpu: 1000m + memory: 2Gi + requests: + cpu: 500m + memory: 1Gi + volumeMounts: + - name: scratch + mountPath: /tmp + - name: federation-config + mountPath: /opt/docker/conf + readOnly: true + volumes: + # readOnlyRootFilesystem=true requires a writable scratch dir: the org.duckdb + # JDBC driver extracts its native libduckdb_java library to java.io.tmpdir (/tmp) + # on every boot (PersistentDuckDB is instantiated unconditionally — even with + # path=":memory:"), and the federation stages CTAS / INSERT-with-JOIN Parquet + # under java.io.tmpdir (FederationConfig.scala:266-267 → /tmp/softclient4es/scratch). + # A file DuckDB path (federation.duckdb.path != ":memory:") would also live here. + - name: scratch + emptyDir: + sizeLimit: 2Gi + - name: federation-config + configMap: + name: fed-softclient4es-federation-config +--- +# Source: softclient4es-federation/templates/sidecar-deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: fed-softclient4es-federation-us-east-1 + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: us-east-1 + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: us-east-1 + template: + metadata: + labels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: us-east-1 + spec: + serviceAccountName: fed-softclient4es-federation + securityContext: + fsGroup: 1001 + runAsGroup: 1001 + runAsNonRoot: true + runAsUser: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: sidecar + image: "docker.io/softnetwork/softclient4es8-arrow-flight-sql:0.2.0" + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1001 + ports: + - name: flight-sql + containerPort: 32010 + protocol: TCP + env: + # ── Arrow Flight SQL server (arrow.flight.*) ── + - name: ARROW_HOST + value: "0.0.0.0" + - name: ARROW_PORT + value: "32010" + - name: ARROW_BATCH_SIZE + value: "1000" + - name: ARROW_QUERY_TIMEOUT_SECONDS + value: "120" + - name: ARROW_JOIN_MAX_MEMORY + value: "256m" + - name: ARROW_JOIN_UPGRADE_URL + value: "https://portal.softclient4es.com/pricing" + # ── Elasticsearch connection (elastic.credentials.* — VERIFIED, NOT a single URL) ── + - name: ELASTIC_SCHEME + value: "https" + - name: ELASTIC_HOST + value: "es-us-east-1.example.com" + - name: ELASTIC_PORT + value: "9200" + - name: ELASTIC_AUTH_METHOD + valueFrom: + secretKeyRef: + name: sc4es-es-us-east-1 + key: es-auth-method + optional: true + - name: ELASTIC_CREDENTIALS_USERNAME + valueFrom: + secretKeyRef: + name: sc4es-es-us-east-1 + key: es-username + optional: true + - name: ELASTIC_CREDENTIALS_PASSWORD + valueFrom: + secretKeyRef: + name: sc4es-es-us-east-1 + key: es-password + optional: true + - name: ELASTIC_CREDENTIALS_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-es-us-east-1 + key: es-api-key + optional: true + - name: ELASTIC_CREDENTIALS_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sc4es-es-us-east-1 + key: es-bearer-token + optional: true + # ── Sidecar SERVER-SIDE auth (incoming) — single source of truth with the + # federation's outgoing servers..credentials (rendered in the ConfigMap) ── + - name: ARROW_AUTH_METHOD + value: "bearer" + - name: ARROW_AUTH_USERNAME + valueFrom: + secretKeyRef: + name: sc4es-arrow-us-east-1 + key: arrow-username + optional: true + - name: ARROW_AUTH_PASSWORD + valueFrom: + secretKeyRef: + name: sc4es-arrow-us-east-1 + key: arrow-password + optional: true + - name: ARROW_AUTH_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sc4es-arrow-us-east-1 + key: arrow-bearer-token + optional: true + - name: ARROW_AUTH_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-arrow-us-east-1 + key: arrow-api-key + optional: true + # ── Telemetry opt-out (shared with federation) ── + # VERIFIED: SOFTCLIENT4ES_TELEMETRY_ENABLED is honored by the sidecar too — the env + # override lives in elasticsql licensing reference.conf:30 (softclient4es.telemetry.enabled), + # read by TelemetryConfig.load, on which the arrow sidecar depends transitively. NOTE it is + # NOT in arrowServerSettings.dockerEnvVars (build.sbt:243-255 bakes only ELASTIC_*/ARROW_*), + # so the chart MUST set it explicitly here for the opt-out to reach the sidecar. + - name: SOFTCLIENT4ES_TELEMETRY_ENABLED + value: "true" + - name: SOFTCLIENT4ES_LICENSE_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: license-key + optional: true + - name: SOFTCLIENT4ES_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: api-key + optional: true + # TCP probes — the sidecar has NO gRPC health service (VERIFIED); a raw TCP + # connect to ARROW_PORT proves the Flight SQL server is listening. + livenessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 20 + periodSeconds: 15 + failureThreshold: 3 + readinessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + resources: + limits: + cpu: 1000m + memory: 1536Mi + requests: + cpu: 500m + memory: 768Mi + volumeMounts: + - name: scratch + mountPath: /tmp + volumes: + # Writable /tmp: the sidecar runs the same-cluster JOIN engine on DuckDB + # (DuckDBJoinExecutor via LocalJoinHandler — VERIFIED), whose org.duckdb JDBC + # driver extracts its native libduckdb_java to java.io.tmpdir on first JOIN. + # readOnlyRootFilesystem=true without this would crash the first JOIN query. + - name: scratch + emptyDir: + sizeLimit: 1Gi +--- +# Source: softclient4es-federation/templates/sidecar-deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: fed-softclient4es-federation-eu-west-1 + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: eu-west-1 + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: eu-west-1 + template: + metadata: + labels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: eu-west-1 + spec: + serviceAccountName: fed-softclient4es-federation + securityContext: + fsGroup: 1001 + runAsGroup: 1001 + runAsNonRoot: true + runAsUser: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: sidecar + image: "docker.io/softnetwork/softclient4es8-arrow-flight-sql:0.2.0" + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1001 + ports: + - name: flight-sql + containerPort: 32010 + protocol: TCP + env: + # ── Arrow Flight SQL server (arrow.flight.*) ── + - name: ARROW_HOST + value: "0.0.0.0" + - name: ARROW_PORT + value: "32010" + - name: ARROW_BATCH_SIZE + value: "1000" + - name: ARROW_QUERY_TIMEOUT_SECONDS + value: "120" + - name: ARROW_JOIN_MAX_MEMORY + value: "256m" + - name: ARROW_JOIN_UPGRADE_URL + value: "https://portal.softclient4es.com/pricing" + # ── Elasticsearch connection (elastic.credentials.* — VERIFIED, NOT a single URL) ── + - name: ELASTIC_SCHEME + value: "https" + - name: ELASTIC_HOST + value: "es-eu-west-1.example.com" + - name: ELASTIC_PORT + value: "9200" + - name: ELASTIC_AUTH_METHOD + valueFrom: + secretKeyRef: + name: sc4es-es-eu-west-1 + key: es-auth-method + optional: true + - name: ELASTIC_CREDENTIALS_USERNAME + valueFrom: + secretKeyRef: + name: sc4es-es-eu-west-1 + key: es-username + optional: true + - name: ELASTIC_CREDENTIALS_PASSWORD + valueFrom: + secretKeyRef: + name: sc4es-es-eu-west-1 + key: es-password + optional: true + - name: ELASTIC_CREDENTIALS_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-es-eu-west-1 + key: es-api-key + optional: true + - name: ELASTIC_CREDENTIALS_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sc4es-es-eu-west-1 + key: es-bearer-token + optional: true + # ── Sidecar SERVER-SIDE auth (incoming) — single source of truth with the + # federation's outgoing servers..credentials (rendered in the ConfigMap) ── + - name: ARROW_AUTH_METHOD + value: "bearer" + - name: ARROW_AUTH_USERNAME + valueFrom: + secretKeyRef: + name: sc4es-arrow-eu-west-1 + key: arrow-username + optional: true + - name: ARROW_AUTH_PASSWORD + valueFrom: + secretKeyRef: + name: sc4es-arrow-eu-west-1 + key: arrow-password + optional: true + - name: ARROW_AUTH_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sc4es-arrow-eu-west-1 + key: arrow-bearer-token + optional: true + - name: ARROW_AUTH_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-arrow-eu-west-1 + key: arrow-api-key + optional: true + # ── Telemetry opt-out (shared with federation) ── + # VERIFIED: SOFTCLIENT4ES_TELEMETRY_ENABLED is honored by the sidecar too — the env + # override lives in elasticsql licensing reference.conf:30 (softclient4es.telemetry.enabled), + # read by TelemetryConfig.load, on which the arrow sidecar depends transitively. NOTE it is + # NOT in arrowServerSettings.dockerEnvVars (build.sbt:243-255 bakes only ELASTIC_*/ARROW_*), + # so the chart MUST set it explicitly here for the opt-out to reach the sidecar. + - name: SOFTCLIENT4ES_TELEMETRY_ENABLED + value: "true" + - name: SOFTCLIENT4ES_LICENSE_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: license-key + optional: true + - name: SOFTCLIENT4ES_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: api-key + optional: true + # TCP probes — the sidecar has NO gRPC health service (VERIFIED); a raw TCP + # connect to ARROW_PORT proves the Flight SQL server is listening. + livenessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 20 + periodSeconds: 15 + failureThreshold: 3 + readinessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + resources: + limits: + cpu: 1000m + memory: 1536Mi + requests: + cpu: 500m + memory: 768Mi + volumeMounts: + - name: scratch + mountPath: /tmp + volumes: + # Writable /tmp: the sidecar runs the same-cluster JOIN engine on DuckDB + # (DuckDBJoinExecutor via LocalJoinHandler — VERIFIED), whose org.duckdb JDBC + # driver extracts its native libduckdb_java to java.io.tmpdir on first JOIN. + # readOnlyRootFilesystem=true without this would crash the first JOIN query. + - name: scratch + emptyDir: + sizeLimit: 1Gi +--- +# Source: softclient4es-federation/templates/sidecar-deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: fed-softclient4es-federation-ap-south-1 + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: ap-south-1 + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: softclient4es +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: ap-south-1 + template: + metadata: + labels: + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/component: sidecar + softclient4es.app/sidecar: ap-south-1 + spec: + serviceAccountName: fed-softclient4es-federation + securityContext: + fsGroup: 1001 + runAsGroup: 1001 + runAsNonRoot: true + runAsUser: 1001 + seccompProfile: + type: RuntimeDefault + containers: + - name: sidecar + image: "docker.io/softnetwork/softclient4es9-arrow-flight-sql:0.2.0" + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1001 + ports: + - name: flight-sql + containerPort: 32010 + protocol: TCP + env: + # ── Arrow Flight SQL server (arrow.flight.*) ── + - name: ARROW_HOST + value: "0.0.0.0" + - name: ARROW_PORT + value: "32010" + - name: ARROW_BATCH_SIZE + value: "1000" + - name: ARROW_QUERY_TIMEOUT_SECONDS + value: "120" + - name: ARROW_JOIN_MAX_MEMORY + value: "256m" + - name: ARROW_JOIN_UPGRADE_URL + value: "https://portal.softclient4es.com/pricing" + # ── Elasticsearch connection (elastic.credentials.* — VERIFIED, NOT a single URL) ── + - name: ELASTIC_SCHEME + value: "https" + - name: ELASTIC_HOST + value: "es-ap-south-1.example.com" + - name: ELASTIC_PORT + value: "9200" + - name: ELASTIC_AUTH_METHOD + valueFrom: + secretKeyRef: + name: sc4es-es-ap-south-1 + key: es-auth-method + optional: true + - name: ELASTIC_CREDENTIALS_USERNAME + valueFrom: + secretKeyRef: + name: sc4es-es-ap-south-1 + key: es-username + optional: true + - name: ELASTIC_CREDENTIALS_PASSWORD + valueFrom: + secretKeyRef: + name: sc4es-es-ap-south-1 + key: es-password + optional: true + - name: ELASTIC_CREDENTIALS_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-es-ap-south-1 + key: es-api-key + optional: true + - name: ELASTIC_CREDENTIALS_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sc4es-es-ap-south-1 + key: es-bearer-token + optional: true + # ── Sidecar SERVER-SIDE auth (incoming) — single source of truth with the + # federation's outgoing servers..credentials (rendered in the ConfigMap) ── + - name: ARROW_AUTH_METHOD + value: "bearer" + - name: ARROW_AUTH_USERNAME + valueFrom: + secretKeyRef: + name: sc4es-arrow-ap-south-1 + key: arrow-username + optional: true + - name: ARROW_AUTH_PASSWORD + valueFrom: + secretKeyRef: + name: sc4es-arrow-ap-south-1 + key: arrow-password + optional: true + - name: ARROW_AUTH_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sc4es-arrow-ap-south-1 + key: arrow-bearer-token + optional: true + - name: ARROW_AUTH_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-arrow-ap-south-1 + key: arrow-api-key + optional: true + # ── Telemetry opt-out (shared with federation) ── + # VERIFIED: SOFTCLIENT4ES_TELEMETRY_ENABLED is honored by the sidecar too — the env + # override lives in elasticsql licensing reference.conf:30 (softclient4es.telemetry.enabled), + # read by TelemetryConfig.load, on which the arrow sidecar depends transitively. NOTE it is + # NOT in arrowServerSettings.dockerEnvVars (build.sbt:243-255 bakes only ELASTIC_*/ARROW_*), + # so the chart MUST set it explicitly here for the opt-out to reach the sidecar. + - name: SOFTCLIENT4ES_TELEMETRY_ENABLED + value: "true" + - name: SOFTCLIENT4ES_LICENSE_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: license-key + optional: true + - name: SOFTCLIENT4ES_API_KEY + valueFrom: + secretKeyRef: + name: sc4es-pro-license + key: api-key + optional: true + # TCP probes — the sidecar has NO gRPC health service (VERIFIED); a raw TCP + # connect to ARROW_PORT proves the Flight SQL server is listening. + livenessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 20 + periodSeconds: 15 + failureThreshold: 3 + readinessProbe: + tcpSocket: + port: flight-sql + initialDelaySeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + resources: + limits: + cpu: 1000m + memory: 1536Mi + requests: + cpu: 500m + memory: 768Mi + volumeMounts: + - name: scratch + mountPath: /tmp + volumes: + # Writable /tmp: the sidecar runs the same-cluster JOIN engine on DuckDB + # (DuckDBJoinExecutor via LocalJoinHandler — VERIFIED), whose org.duckdb JDBC + # driver extracts its native libduckdb_java to java.io.tmpdir on first JOIN. + # readOnlyRootFilesystem=true without this would crash the first JOIN query. + - name: scratch + emptyDir: + sizeLimit: 1Gi +--- +# Source: softclient4es-federation/templates/tests/smoke-test.yaml +apiVersion: v1 +kind: Pod +metadata: + name: fed-softclient4es-federation-test-catalogs + labels: + helm.sh/chart: softclient4es-federation-0.3.0 + app.kubernetes.io/name: softclient4es-federation + app.kubernetes.io/instance: fed + app.kubernetes.io/version: "0.2.0" + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/component: federation + app.kubernetes.io/part-of: softclient4es + annotations: + "helm.sh/hook": test + "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded +spec: + restartPolicy: Never + containers: + - name: show-catalogs + image: "python:3.12-slim" + command: ["sh", "-c"] + args: + - | + pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + python - <<'PY' + import sys + import adbc_driver_flightsql.dbapi as dbapi + host = "fed-softclient4es-federation" + conn = dbapi.connect(f"grpc://{host}:32020") + # GetCatalogs: one catalog per registered sidecar alias. + cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + expected = 3 + print("catalogs:", cats) + if len(cats) != expected: + print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) + sys.exit(1) + print("OK") + PY From 51424bea15a0ae424d23f9bcd1146f4e8cdb7244 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Manciot?= Date: Tue, 23 Jun 2026 10:37:20 +0200 Subject: [PATCH 2/3] =?UTF-8?q?Story=2016.5=20=E2=80=94=20helm=20test=20sm?= =?UTF-8?q?oke=20+=20comprehensive=20CI/CD?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the chart-repo CI workflow (.github/workflows/federation-helm.yml) plus the deploy-es.sh / install-sealed-secrets.sh scripts, and the helm test smoke Job. CI redesigned for the chart-only repo: the sbt build-images job is replaced by an image-availability probe that docker-pulls the PUBLIC DockerHub federation + sidecar images and kind-loads them; live-install jobs are gated on image publication and skip-with-warning until published (OQ-1). JFROG creds + setup-java/setup-sbt removed. All static-validation gates (lint, template+kubeconform, golden diff, zero kind:Secret, duckdb-attach discriminator) run unconditionally. Trigger paths updated to softclient4es-federation/**. Smoke = ADBC GetCatalogs (1/3/3 per example). README CI section + Chart.yaml metadata rewritten to match (softclient4es.dev, softclient4es-helm sources, public DockerHub images, private-repo references removed). Closes #9 Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/scripts/deploy-es.sh | 132 +++++ .github/scripts/install-sealed-secrets.sh | 67 +++ .github/workflows/federation-helm.yml | 519 ++++++++++++++++++ softclient4es-federation/README.md | 99 +++- .../templates/deployment.yaml | 13 + .../templates/tests/smoke-test.yaml | 42 +- .../golden/example-heterogeneous-ready.yaml | 42 +- .../tests/golden/example-single-cluster.yaml | 42 +- .../tests/golden/example-three-region.yaml | 42 +- .../tests/golden/ingress-tls.yaml | 42 +- .../tests/golden/secret-auth.yaml | 42 +- .../tests/golden/two-sidecars.yaml | 42 +- softclient4es-federation/values.yaml | 19 +- 13 files changed, 1062 insertions(+), 81 deletions(-) create mode 100755 .github/scripts/deploy-es.sh create mode 100755 .github/scripts/install-sealed-secrets.sh create mode 100644 .github/workflows/federation-helm.yml diff --git a/.github/scripts/deploy-es.sh b/.github/scripts/deploy-es.sh new file mode 100755 index 0000000..2b13c62 --- /dev/null +++ b/.github/scripts/deploy-es.sh @@ -0,0 +1,132 @@ +#!/usr/bin/env bash +# Story 16.5 — deploy single-node Elasticsearch container(s) into a kind cluster, one per +# federation sidecar, reachable at the in-cluster Service name the install job overrides each +# sidecars[].elasticsearch.url to. Used by .github/workflows/federation-helm.yml. +# +# Usage: deploy-es.sh +# topology : single-cluster | three-region | heterogeneous-ready | es-version | secrets +# es-version-list : comma-separated ES major versions, e.g. "8" or "8,8,9" +# +# Service names (must match the --set sidecars[N].elasticsearch.url=... in the install jobs): +# * 1-ES topologies (single-cluster / es-version / secrets) -> Service "es" +# * 3-ES topologies (three-region / heterogeneous-ready) -> "es-us-east-1", "es-eu-west-1", +# "es-ap-south-1" (order = list order) +# +# Each ES runs single-node, security disabled, small heap (fits the 7 GB GitHub runner). +# `vm.max_map_count=262144` MUST already be set on the runner (the workflow does this). +set -euo pipefail + +TOPOLOGY="${1:?usage: deploy-es.sh }" +ES_LIST="${2:?usage: deploy-es.sh }" + +# ES major version -> full image coordinate (CLAUDE.md ES Version Matrix). +es_image() { + case "$1" in + 6) echo "docker.elastic.co/elasticsearch/elasticsearch:6.8.23" ;; + 7) echo "docker.elastic.co/elasticsearch/elasticsearch:7.17.29" ;; + 8) echo "docker.elastic.co/elasticsearch/elasticsearch:8.18.3" ;; + 9) echo "docker.elastic.co/elasticsearch/elasticsearch:9.0.3" ;; + *) echo "unsupported ES version: $1" >&2; exit 1 ;; + esac +} + +# Resolve the Service names for this topology, positionally aligned with the ES list. +case "$TOPOLOGY" in + three-region|heterogeneous-ready) + NAMES=(es-us-east-1 es-eu-west-1 es-ap-south-1) + ;; + single-cluster|es-version|secrets) + NAMES=(es) + ;; + *) + echo "unsupported topology: $TOPOLOGY" >&2; exit 1 ;; +esac + +# Split the comma-separated version list into an array. +IFS=',' read -r -a VERSIONS <<< "$ES_LIST" + +if [ "${#VERSIONS[@]}" -gt "${#NAMES[@]}" ]; then + echo "::error::deploy-es.sh: ${#VERSIONS[@]} ES versions but only ${#NAMES[@]} service name(s) for topology $TOPOLOGY" >&2 + exit 1 +fi + +deploy_one() { + local name="$1" version="$2" image + image="$(es_image "$version")" + echo "Deploying Elasticsearch ${version} as Service '${name}' (${image})" + kubectl apply -f - < +# secret-name : the name of the Secret the controller will materialize (chart references this +# via sidecars[0].elasticsearch.credentialsSecretName), e.g. "es-creds". +# +# Why re-seal in CI (project_sealed_secrets_review_gotchas): +# * The committed examples/sealed-secrets/*.yaml are NON-FUNCTIONAL placeholders — a SealedSecret +# is decryptable ONLY by the specific controller instance that issued the cert. We therefore +# create a fresh plaintext Secret, `kubeseal` it with the live cluster's cert, and apply. +# * The kubeseal CLI version is pinned to the controller chart appVersion (NOT the chart version). +set -euo pipefail + +SECRET_NAME="${1:?usage: install-sealed-secrets.sh }" + +# Pin the controller chart + kubeseal CLI so the CLI/controller versions match (gotcha: +# kubeseal CLI version must equal the controller appVersion, not the Helm chart version). +SEALED_SECRETS_CHART_VERSION="2.16.2" # helm chart version +KUBESEAL_VERSION="0.27.2" # == controller appVersion of chart 2.16.2 + +# 1. Install the controller (repo moved bitnami-labs -> bitnami.github.io). +helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets >/dev/null 2>&1 || \ + helm repo add sealed-secrets https://bitnami.github.io/sealed-secrets +helm repo update sealed-secrets +helm upgrade --install sealed-secrets sealed-secrets/sealed-secrets \ + --namespace kube-system \ + --version "${SEALED_SECRETS_CHART_VERSION}" \ + --set fullnameOverride=sealed-secrets-controller \ + --wait --timeout 180s + +# 2. Install the matching kubeseal CLI. +curl -sSL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz" \ + | tar -xz kubeseal +sudo install -m 0755 kubeseal /usr/local/bin/kubeseal +rm -f kubeseal + +# 3. Wait for the controller to be Ready, then fetch its public cert. +kubectl rollout status deployment/sealed-secrets-controller -n kube-system --timeout=180s +kubeseal --controller-name=sealed-secrets-controller --controller-namespace=kube-system \ + --fetch-cert > /tmp/sealed-secrets-cert.pem + +# 4. Build a plaintext ES-auth Secret (matching the 16.3 es-* key contract), seal it with the +# live cert, and apply the SealedSecret (the plaintext Secret is NEVER applied to the cluster). +kubectl create secret generic "${SECRET_NAME}" \ + --dry-run=client -o yaml \ + --from-literal=es-auth-method=basic \ + --from-literal=es-username=elastic \ + --from-literal=es-password=changeme \ + | kubeseal --cert /tmp/sealed-secrets-cert.pem --format yaml \ + | kubectl apply -f - + +# 5. Wait for the controller to materialize the real Secret from the SealedSecret. +for _ in $(seq 1 30); do + if kubectl get secret "${SECRET_NAME}" >/dev/null 2>&1; then + echo "SealedSecrets materialized Secret '${SECRET_NAME}'." + exit 0 + fi + sleep 2 +done + +echo "::error::SealedSecrets controller did not materialize Secret '${SECRET_NAME}' in time" >&2 +kubectl get sealedsecret,secret 2>/dev/null || true +exit 1 diff --git a/.github/workflows/federation-helm.yml b/.github/workflows/federation-helm.yml new file mode 100644 index 0000000..1c39bff --- /dev/null +++ b/.github/workflows/federation-helm.yml @@ -0,0 +1,519 @@ +# Comprehensive CI for the softclient4es-federation Helm chart (Epic 16, Story 16.5). +# Validates: lint + template + kubeconform + golden-diff (static, hard blocker) AND +# per-topology / per-ES-version / secret-backend / upgrade / uninstall installs on +# ephemeral kind clusters. +# +# This is the dedicated CHART repo (softclient4es-helm) — it is chart-only and CANNOT +# build images via sbt. CI therefore NEVER builds: the live-install jobs `docker pull` +# the PUBLIC DockerHub federation + sidecar images and `kind load` them. checkout@v4, +# `sudo sysctl -w vm.max_map_count=262144` before any Elasticsearch boots. +# +# IMAGE TAGS: the chart's image refs default to the Chart.yaml appVersion. The live jobs +# pull the public DockerHub tags (federation image.tag:"" -> appVersion; sidecar tag -> +# appVersion) and `kind load` them — no per-image --set override, so the federation + +# sidecar tags stay consistent. The committed goldens are appVersion renders, so the +# golden gate renders with NO --set image.tag. +# +# IMAGE AVAILABILITY (OQ-1): neither the federation image nor the sidecar tags are +# published on DockerHub yet. The `image-availability` job probes each with +# `docker manifest inspect`; if ANY required image is absent every live-install job is +# SKIPPED WITH A `::warning::` ANNOTATION (not failed). All static-validation gates run +# UNCONDITIONALLY on every PR with zero external deps. +# +# LICENSE: the multi-sidecar (Pro) tiers are BEST-EFFORT, gated on TWO repo secrets +# together (skip-not-fail if either missing), on top of image availability: (1) secret +# SC4ES_PRO_TEST_JWT, (2) secret SC4ES_TEST_PUBLIC_KEY (injected as +# SOFTCLIENT4ES_LICENSE_PUBLIC_KEY so the verifier resolves the test kid OFFLINE). The +# published federation image must itself be Pro-capable (JWT-verifying SPI on classpath, +# Story 16.1 OQ-5) for the Pro JWT to verify — an OSS-only image falls back to Community. +# single-cluster + per-ES-version + secret backends + uninstall + ALL static gates are +# license-free and run unconditionally (when the images are published). +name: Federation Helm Chart + +on: + workflow_dispatch: + pull_request: + paths: + - 'softclient4es-federation/**' + - '.github/workflows/federation-helm.yml' + - '.github/scripts/deploy-es.sh' + - '.github/scripts/install-sealed-secrets.sh' + push: + branches: ['!main'] + paths: + - 'softclient4es-federation/**' + - '.github/workflows/federation-helm.yml' + - '.github/scripts/deploy-es.sh' + - '.github/scripts/install-sealed-secrets.sh' + # Heavy matrix (per-ES-version + secret backends) can be moved here if PR duration > 30 min + # (see the sharding fallback in the chart README). PRs would then run a reduced matrix. + # schedule: + # - cron: '0 3 * * *' + +permissions: + contents: read + +env: + CHART_DIR: softclient4es-federation + HELM_VERSION: v3.16.3 + KUBECONFORM_VERSION: v0.6.7 + KIND_VERSION: v0.24.0 # kind node images current to K8s 1.31 + KIND_NODE_IMAGE: kindest/node:v1.31.0 + # The chart's image refs default to the Chart.yaml appVersion. The live jobs pull the + # public DockerHub tags at this version so the chart DEFAULTS resolve the kind-loaded images. + IMAGE_TAG: '0.2.0' + FED_IMAGE: softnetwork/softclient4es-federation + SIDECAR_IMAGE_PREFIX: softnetwork/softclient4es # softclient4es-arrow-flight-sql + # A pre-baked ADBC client image avoids the smoke Job's runtime pip install (air-gap). When + # empty the chart smoke Pod falls back to python:3.12-slim + runtime pip (PyPI reachable on + # GitHub-hosted runners). PROPOSED image — see OQ-2 / chart README. + TEST_ADBC_IMAGE: '' + +jobs: + # ── 1. STATIC CHECKS — no cluster, unconditional, hard blocker ───────────── + static-checks: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Set up Helm + uses: azure/setup-helm@v4 + with: + version: ${{ env.HELM_VERSION }} + - name: Install kubeconform + run: | + curl -sSL "https://github.com/yannh/kubeconform/releases/download/${KUBECONFORM_VERSION}/kubeconform-linux-amd64.tar.gz" \ + | sudo tar -xz -C /usr/local/bin kubeconform + - name: helm lint (chart + all examples) + run: | + helm lint "${CHART_DIR}" + for ex in single-cluster three-region heterogeneous-ready; do + f="${CHART_DIR}/examples/${ex}/values.yaml" + if [ -f "$f" ]; then + helm lint "${CHART_DIR}" -f "$f" + else + echo "::warning::missing example $f (Story 16.4 not merged yet — skipping lint -f)" + fi + done + - name: helm template + kubeconform (base + each example) + run: | + render() { helm template fed "${CHART_DIR}" "$@"; } + # base (0-sidecar) always renders + render | kubeconform -strict -summary -kubernetes-version 1.29.0 -ignore-missing-schemas + for ex in single-cluster three-region heterogeneous-ready; do + f="${CHART_DIR}/examples/${ex}/values.yaml" + if [ -f "$f" ]; then + render -f "$f" | kubeconform -strict -summary -kubernetes-version 1.29.0 -ignore-missing-schemas + else + echo "::warning::missing example $f — skipping kubeconform for $ex" + fi + done + - name: golden-file diff (template drift gate) + run: | + fail=0 + # default (16.1) + two-sidecars (16.2) + secret-auth/ingress-tls (16.3) goldens. + # All committed goldens are appVersion renders — render WITH NO --set image.tag. + helm template fed "${CHART_DIR}" \ + | diff -u "${CHART_DIR}/tests/golden/default.yaml" - || { echo "::error::default.yaml golden drift"; fail=1; } + helm template fed "${CHART_DIR}" -f "${CHART_DIR}/tests/values/two-sidecars.yaml" \ + | diff -u "${CHART_DIR}/tests/golden/two-sidecars.yaml" - || { echo "::error::two-sidecars.yaml golden drift"; fail=1; } + helm template fed "${CHART_DIR}" -f "${CHART_DIR}/tests/values/secret-auth.yaml" \ + | diff -u "${CHART_DIR}/tests/golden/secret-auth.yaml" - || { echo "::error::secret-auth.yaml golden drift"; fail=1; } + helm template fed "${CHART_DIR}" -f "${CHART_DIR}/tests/values/ingress-tls.yaml" \ + | diff -u "${CHART_DIR}/tests/golden/ingress-tls.yaml" - || { echo "::error::ingress-tls.yaml golden drift"; fail=1; } + # Per-example goldens (Story 16.4 committed them at tests/golden/example-.yaml). + for ex in single-cluster three-region heterogeneous-ready; do + f="${CHART_DIR}/examples/${ex}/values.yaml" + g="${CHART_DIR}/tests/golden/example-${ex}.yaml" + if [ -f "$f" ] && [ -f "$g" ]; then + helm template fed "${CHART_DIR}" -f "$f" \ + | diff -u "$g" - || { echo "::error::example-${ex}.yaml golden drift"; fail=1; } + else + echo "::warning::missing example or golden for $ex (16.4 not merged) — skipping golden diff" + fi + done + exit $fail + - name: assert chart renders ZERO kind:Secret (16.3 contract) + run: | + n=$(helm template fed "${CHART_DIR}" | grep -c '^kind: Secret' || true) + [ "$n" = "0" ] || { echo "::error::chart must NOT render a kind:Secret (got $n)"; exit 1; } + - name: heterogeneous-ready discriminator (16.4 A2b — only signal vs three-region) + run: | + # three-region and heterogeneous-ready goldens are BYTE-IDENTICAL by design (Helm + # strips the R2b duckdb-attach comment block). The ONLY signal distinguishing them is + # the commented duckdb-attach preview in the SOURCE values.yaml. Guard against a + # regression that copies one example over the other (passes golden + smoke undetected). + het="${CHART_DIR}/examples/heterogeneous-ready/values.yaml" + tr="${CHART_DIR}/examples/three-region/values.yaml" + if [ -f "$het" ] && [ -f "$tr" ]; then + nh=$(grep -c 'type = "duckdb-attach"' "$het" || true) + nt=$(grep -c 'type = "duckdb-attach"' "$tr" || true) + [ "$nh" = "3" ] || { echo "::error::heterogeneous-ready must keep 3 commented duckdb-attach previews (got $nh)"; exit 1; } + [ "$nt" = "0" ] || { echo "::error::three-region must have 0 duckdb-attach entries (got $nt)"; exit 1; } + else + echo "::warning::examples missing (16.4 not merged) — skipping A2b discriminator" + fi + + # ── 2. IMAGE AVAILABILITY — probe the PUBLIC DockerHub images, prerequisite for installs ─ + # This chart repo cannot build images (no sbt). The federation image + the four sidecar tags + # are pulled from public DockerHub. They are NOT yet published (OQ-1), so probe each tag with + # `docker manifest inspect`; if ANY is absent, the live-install jobs SKIP WITH A WARNING. + image-availability: + runs-on: ubuntu-latest + outputs: + images_published: ${{ steps.probe.outputs.images_published }} # gate for ALL live-install jobs + steps: + - name: Probe public DockerHub images (federation + 4 sidecars) + id: probe + run: | + published=true + probe() { + if docker manifest inspect "$1" >/dev/null 2>&1; then + echo "found: $1" + else + echo "::warning::image not published on DockerHub: $1 — live-install jobs will be SKIPPED (OQ-1)" + published=false + fi + } + probe "docker.io/${FED_IMAGE}:${IMAGE_TAG}" + for v in 6 7 8 9; do + probe "docker.io/${SIDECAR_IMAGE_PREFIX}${v}-arrow-flight-sql:${IMAGE_TAG}" + done + if [ "$published" != "true" ]; then + echo "::warning::one or more federation/sidecar images are not yet published (OQ-1) — skipping all live kind-install jobs; static gates still ran" + fi + echo "images_published=$published" >> "$GITHUB_OUTPUT" + + # ── 3. PER-TOPOLOGY INSTALL — single-cluster (license-free) + 3-cluster (Pro) ─ + install-topology: + runs-on: ubuntu-latest + needs: [static-checks, image-availability] + if: ${{ needs.image-availability.outputs.images_published == 'true' }} + strategy: + fail-fast: false + matrix: + include: + - example: single-cluster + es: '8' # one ES8, hostname `es` + expect_catalogs: 1 + license: false # 1 sidecar -> Community, no license (FACT C) + - example: three-region + es: '8,8,9' # two ES8 + one ES9 (mixed-version) + expect_catalogs: 3 + license: true # 3 sidecars -> Pro JWT required (FACT C/F) + - example: heterogeneous-ready + es: '8,8,9' # 3 ES (R2b placeholders inactive in R1) + expect_catalogs: 3 + license: true + steps: + - uses: actions/checkout@v4 + - name: Set vm.max_map_count (Elasticsearch requirement) + run: sudo sysctl -w vm.max_map_count=262144 + - name: Skip if example missing (Story 16.4 not merged) + id: guard + run: | + f="${CHART_DIR}/examples/${{ matrix.example }}/values.yaml" + if [ ! -f "$f" ]; then + echo "::warning::$f missing — Story 16.4 not merged; skipping install" + echo "skip=true" >> "$GITHUB_OUTPUT" + fi + - name: Skip multi-sidecar install when no Pro test license (FACT C + FACT F, best-effort tier) + id: lic + if: ${{ matrix.license && steps.guard.outputs.skip != 'true' }} + # The 3-sidecar tier needs BOTH: a Pro test JWT and its public verification key (so the + # verifier resolves the test kid OFFLINE). Missing EITHER → SKIP (annotation), never fail. + # (The published federation image must itself be Pro-capable — 16.1 OQ-5 — for the JWT to + # verify; an OSS-only image silently falls back to Community and CrashLoops at 3 clusters.) + run: | + if [ -z "${{ secrets.SC4ES_PRO_TEST_JWT }}" ] || [ -z "${{ secrets.SC4ES_TEST_PUBLIC_KEY }}" ]; then + echo "::warning::missing SC4ES_PRO_TEST_JWT and/or SC4ES_TEST_PUBLIC_KEY — ${{ matrix.example }} (3 sidecars) needs a Pro JWT + its public key (FACT F #2); skipping (best-effort)" + echo "skip=true" >> "$GITHUB_OUTPUT" + fi + - name: Set up Helm + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + uses: azure/setup-helm@v4 + with: { version: "${{ env.HELM_VERSION }}" } + - name: Create kind cluster + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + uses: helm/kind-action@v1.10.0 + with: + version: ${{ env.KIND_VERSION }} + node_image: ${{ env.KIND_NODE_IMAGE }} + cluster_name: fed-${{ matrix.example }} + - name: Pull public images + kind load (federation + sidecars) + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + run: | + docker pull "docker.io/${FED_IMAGE}:${IMAGE_TAG}" + kind load docker-image "docker.io/${FED_IMAGE}:${IMAGE_TAG}" --name "fed-${{ matrix.example }}" + for v in $(echo '${{ matrix.es }}' | tr ',' '\n' | sort -u); do + img="docker.io/${SIDECAR_IMAGE_PREFIX}${v}-arrow-flight-sql:${IMAGE_TAG}" + docker pull "$img" + kind load docker-image "$img" --name "fed-${{ matrix.example }}" + done + - name: Deploy ES container(s) per sidecar (FACT D — reachable ES required) + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + run: ./.github/scripts/deploy-es.sh "${{ matrix.example }}" '${{ matrix.es }}' + - name: Create Pro license + public-key Secret (multi-sidecar only — FACT F #2) + if: ${{ matrix.license && steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + run: | + # ONE Secret carries both the JWT (license-key) and the public verification JWK + # (license-public-key → SOFTCLIENT4ES_LICENSE_PUBLIC_KEY, the air-gap path in + # LicenseKeyVerifier.loadPublicKey Step 3 — without it the test kid never resolves). + kubectl create secret generic sc4es-license \ + --from-literal=license-key="${{ secrets.SC4ES_PRO_TEST_JWT }}" \ + --from-literal=license-public-key="${{ secrets.SC4ES_TEST_PUBLIC_KEY }}" + # The three-region example pins per-region license.secretName=sc4es-pro-license; also + # create that name so its bearer-auth sidecars + the federation share the test token. + kubectl create secret generic sc4es-pro-license \ + --from-literal=license-key="${{ secrets.SC4ES_PRO_TEST_JWT }}" \ + --from-literal=license-public-key="${{ secrets.SC4ES_TEST_PUBLIC_KEY }}" || true + - name: helm install + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + run: | + # The example values point sidecars[].elasticsearch.url at external https://*.example.com + # placeholders. deploy-es.sh stands up in-cluster ES Services; override each URL to the + # matching plaintext in-cluster Service. license/public-key are pointed at sc4es-license + # for the multi-sidecar tier. + extra="" + if [ "${{ matrix.license }}" = "true" ]; then + extra="--set license.secretName=sc4es-license --set license.publicKeySecretName=sc4es-license" + fi + # The three-region/heterogeneous examples set sidecars[].auth.method=bearer pointing at + # per-region sc4es-arrow-* Secrets that this CI does NOT create (the dedicated + # install-secrets job covers Secret-backed auth). Override the federation↔sidecar hop to + # auth.method=none so catalog discovery is what's under test here, not bearer auth — else + # the sidecars enforce bearer with an absent (optional) token and discovery fails. + urls="" + case "${{ matrix.example }}" in + single-cluster) + urls="--set sidecars[0].elasticsearch.url=http://es:9200" ;; + three-region|heterogeneous-ready) + urls="--set sidecars[0].elasticsearch.url=http://es-us-east-1:9200 \ + --set sidecars[1].elasticsearch.url=http://es-eu-west-1:9200 \ + --set sidecars[2].elasticsearch.url=http://es-ap-south-1:9200 \ + --set sidecars[0].auth.method=none \ + --set sidecars[1].auth.method=none \ + --set sidecars[2].auth.method=none" ;; + esac + helm install fed "${CHART_DIR}" \ + -f "${CHART_DIR}/examples/${{ matrix.example }}/values.yaml" \ + ${TEST_ADBC_IMAGE:+--set test.image=$TEST_ADBC_IMAGE} \ + $urls $extra + - name: Wait all pods Ready (sidecars + federation — FACT D) + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + run: kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=softclient4es --timeout=300s + - name: helm test (GetCatalogs == expected — FACT A) + if: ${{ steps.guard.outputs.skip != 'true' && steps.lic.outputs.skip != 'true' }} + run: helm test fed --timeout 180s + - name: Diagnostics on failure + if: ${{ failure() }} + run: | + kubectl get pods -o wide || true + kubectl logs -l app.kubernetes.io/component=federation --tail=200 || true + kubectl logs -l app.kubernetes.io/component=sidecar --tail=100 || true + kubectl logs -l app.kubernetes.io/name=softclient4es-federation -c show-catalogs --tail=100 || true + + # ── 4. PER-ES-VERSION SIDECAR MATRIX — 1 sidecar each (license-free) ──────── + install-es-version: + runs-on: ubuntu-latest + needs: [static-checks, image-availability] + if: ${{ needs.image-availability.outputs.images_published == 'true' }} + strategy: + fail-fast: false + matrix: + es: [6, 7, 8, 9] + steps: + - uses: actions/checkout@v4 + - name: Set vm.max_map_count (Elasticsearch requirement) + run: sudo sysctl -w vm.max_map_count=262144 + - name: Set up Helm + uses: azure/setup-helm@v4 + with: { version: "${{ env.HELM_VERSION }}" } + - name: Create kind cluster + uses: helm/kind-action@v1.10.0 + with: { version: "${{ env.KIND_VERSION }}", node_image: "${{ env.KIND_NODE_IMAGE }}", cluster_name: "fed-es${{ matrix.es }}" } + - name: Pull public images + kind load (federation + ES${{ matrix.es }} sidecar) + run: | + docker pull "docker.io/${FED_IMAGE}:${IMAGE_TAG}" + kind load docker-image "docker.io/${FED_IMAGE}:${IMAGE_TAG}" --name "fed-es${{ matrix.es }}" + img="docker.io/${SIDECAR_IMAGE_PREFIX}${{ matrix.es }}-arrow-flight-sql:${IMAGE_TAG}" + docker pull "$img" + kind load docker-image "$img" --name "fed-es${{ matrix.es }}" + - name: Deploy ES ${{ matrix.es }} + run: ./.github/scripts/deploy-es.sh es-version '${{ matrix.es }}' + - name: helm install (1 sidecar, ES ${{ matrix.es }}, license-free) + run: | + helm install fed "${CHART_DIR}" \ + ${TEST_ADBC_IMAGE:+--set test.image=$TEST_ADBC_IMAGE} \ + --set 'sidecars[0].name=primary' \ + --set "sidecars[0].elasticsearchVersion=${{ matrix.es }}" \ + --set 'sidecars[0].elasticsearch.url=http://es:9200' \ + --set 'sidecars[0].default=true' + - name: Wait Ready + helm test (== 1) + run: | + kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=softclient4es --timeout=300s + helm test fed --timeout 180s + - name: Diagnostics on failure + if: ${{ failure() }} + run: | + kubectl get pods -o wide || true + kubectl logs -l app.kubernetes.io/component=federation --tail=200 || true + kubectl logs -l app.kubernetes.io/component=sidecar --tail=100 || true + + # ── 5. SECRET MANAGEMENT — raw Secret + SealedSecrets (1 sidecar, license-free) ─ + install-secrets: + runs-on: ubuntu-latest + needs: [static-checks, image-availability] + if: ${{ needs.image-availability.outputs.images_published == 'true' }} + strategy: + fail-fast: false + matrix: + backend: [raw-secret, sealed-secrets] + steps: + - uses: actions/checkout@v4 + - name: Set vm.max_map_count (Elasticsearch requirement) + run: sudo sysctl -w vm.max_map_count=262144 + - name: Set up Helm + uses: azure/setup-helm@v4 + with: { version: "${{ env.HELM_VERSION }}" } + - name: Create kind cluster + uses: helm/kind-action@v1.10.0 + with: { version: "${{ env.KIND_VERSION }}", node_image: "${{ env.KIND_NODE_IMAGE }}", cluster_name: "fed-${{ matrix.backend }}" } + - name: Pull public images + kind load (federation + ES8 sidecar) + run: | + docker pull "docker.io/${FED_IMAGE}:${IMAGE_TAG}" + kind load docker-image "docker.io/${FED_IMAGE}:${IMAGE_TAG}" --name "fed-${{ matrix.backend }}" + img="docker.io/${SIDECAR_IMAGE_PREFIX}8-arrow-flight-sql:${IMAGE_TAG}" + docker pull "$img" + kind load docker-image "$img" --name "fed-${{ matrix.backend }}" + - name: Deploy ES 8 + run: ./.github/scripts/deploy-es.sh secrets '8' + - name: Install SealedSecrets controller + materialize the ES Secret (sealed-secrets only) + if: ${{ matrix.backend == 'sealed-secrets' }} + run: ./.github/scripts/install-sealed-secrets.sh es-creds + - name: Create the ES auth Secret (raw-secret only) + if: ${{ matrix.backend == 'raw-secret' }} + run: | + kubectl create secret generic es-creds \ + --from-literal=es-auth-method=basic \ + --from-literal=es-username=elastic --from-literal=es-password=changeme + - name: helm install (1 sidecar, Secret-backed ES auth) + run: | + helm install fed "${CHART_DIR}" \ + ${TEST_ADBC_IMAGE:+--set test.image=$TEST_ADBC_IMAGE} \ + --set 'sidecars[0].name=primary' --set 'sidecars[0].elasticsearchVersion=8' \ + --set 'sidecars[0].elasticsearch.url=http://es:9200' \ + --set 'sidecars[0].elasticsearch.credentialsSecretName=es-creds' \ + --set 'sidecars[0].default=true' + - name: Wait Ready + assert ES env injected + helm test + run: | + kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=softclient4es --timeout=300s + # Assert the Secret-backed env actually reached the sidecar container (16.3 AC9). + kubectl get pod -l app.kubernetes.io/component=sidecar \ + -o jsonpath='{.items[0].spec.containers[0].env[*].name}' | grep -q ELASTIC_CREDENTIALS_USERNAME + helm test fed --timeout 180s + - name: Diagnostics on failure + if: ${{ failure() }} + run: | + kubectl get pods -o wide || true + kubectl logs -l app.kubernetes.io/component=federation --tail=200 || true + kubectl logs -l app.kubernetes.io/component=sidecar --tail=100 || true + + # ── 6. UPGRADE — single-cluster -> three-region (Pro license, best-effort) ── + upgrade: + runs-on: ubuntu-latest + needs: [static-checks, image-availability] + if: ${{ needs.image-availability.outputs.images_published == 'true' }} + steps: + - uses: actions/checkout@v4 + - name: Set vm.max_map_count (Elasticsearch requirement) + run: sudo sysctl -w vm.max_map_count=262144 + - name: Skip if examples missing or no Pro test license (FACT C + FACT F) + id: guard + # Upgrade ends at 3 clusters → same Pro gate as install-topology: JWT secret AND + # public-key secret (+ the published image must be Pro-capable, 16.1 OQ-5). + run: | + if [ ! -f "${CHART_DIR}/examples/single-cluster/values.yaml" ] || [ ! -f "${CHART_DIR}/examples/three-region/values.yaml" ] \ + || [ -z "${{ secrets.SC4ES_PRO_TEST_JWT }}" ] || [ -z "${{ secrets.SC4ES_TEST_PUBLIC_KEY }}" ]; then + echo "::warning::upgrade test needs both examples (16.4) + SC4ES_PRO_TEST_JWT + SC4ES_TEST_PUBLIC_KEY (and a Pro-capable image, FACT F); skipping" + echo "skip=true" >> "$GITHUB_OUTPUT" + fi + - name: Set up Helm + if: ${{ steps.guard.outputs.skip != 'true' }} + uses: azure/setup-helm@v4 + with: { version: "${{ env.HELM_VERSION }}" } + - name: Create kind cluster + if: ${{ steps.guard.outputs.skip != 'true' }} + uses: helm/kind-action@v1.10.0 + with: { version: "${{ env.KIND_VERSION }}", node_image: "${{ env.KIND_NODE_IMAGE }}", cluster_name: fed-upgrade } + - name: Pull public images + kind load + ES + license + install single-cluster + if: ${{ steps.guard.outputs.skip != 'true' }} + run: | + docker pull "docker.io/${FED_IMAGE}:${IMAGE_TAG}" + kind load docker-image "docker.io/${FED_IMAGE}:${IMAGE_TAG}" --name fed-upgrade + for v in 8 9; do + img="docker.io/${SIDECAR_IMAGE_PREFIX}${v}-arrow-flight-sql:${IMAGE_TAG}" + docker pull "$img" + kind load docker-image "$img" --name fed-upgrade + done + ./.github/scripts/deploy-es.sh three-region '8,8,9' + # single-cluster needs a `es` Service too (its sidecar URL is overridden to http://es:9200). + ./.github/scripts/deploy-es.sh secrets '8' + for n in sc4es-license sc4es-pro-license; do + kubectl create secret generic "$n" \ + --from-literal=license-key="${{ secrets.SC4ES_PRO_TEST_JWT }}" \ + --from-literal=license-public-key="${{ secrets.SC4ES_TEST_PUBLIC_KEY }}" || true + done + helm install fed "${CHART_DIR}" -f "${CHART_DIR}/examples/single-cluster/values.yaml" \ + ${TEST_ADBC_IMAGE:+--set test.image=$TEST_ADBC_IMAGE} \ + --set sidecars[0].elasticsearch.url=http://es:9200 + kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=softclient4es --timeout=300s + - name: helm upgrade -> three-region + assert topology change + if: ${{ steps.guard.outputs.skip != 'true' }} + run: | + before=$(kubectl get deploy -l app.kubernetes.io/component=sidecar --no-headers | wc -l) + # Same auth.method=none override as install-topology: this CI tests the topology change + # + catalog count, not the per-region bearer Secrets (which it does not create). + helm upgrade fed "${CHART_DIR}" -f "${CHART_DIR}/examples/three-region/values.yaml" \ + ${TEST_ADBC_IMAGE:+--set test.image=$TEST_ADBC_IMAGE} \ + --set license.secretName=sc4es-license --set license.publicKeySecretName=sc4es-license \ + --set sidecars[0].elasticsearch.url=http://es-us-east-1:9200 \ + --set sidecars[1].elasticsearch.url=http://es-eu-west-1:9200 \ + --set sidecars[2].elasticsearch.url=http://es-ap-south-1:9200 \ + --set sidecars[0].auth.method=none \ + --set sidecars[1].auth.method=none \ + --set sidecars[2].auth.method=none + kubectl wait --for=condition=Ready pod -l app.kubernetes.io/part-of=softclient4es --timeout=300s + after=$(kubectl get deploy -l app.kubernetes.io/component=sidecar --no-headers | wc -l) + [ "$after" -gt "$before" ] || { echo "::error::upgrade did not add sidecar Deployments ($before -> $after)"; exit 1; } + helm test fed --timeout 180s + - name: Diagnostics on failure + if: ${{ failure() && steps.guard.outputs.skip != 'true' }} + run: | + kubectl get pods,deploy -o wide || true + kubectl logs -l app.kubernetes.io/component=federation --tail=200 || true + + # ── 7. UNINSTALL — clean teardown, no leftovers (license-free, 0 sidecars) ── + uninstall: + runs-on: ubuntu-latest + needs: [static-checks, image-availability] + if: ${{ needs.image-availability.outputs.images_published == 'true' }} + steps: + - uses: actions/checkout@v4 + - name: Set up Helm + uses: azure/setup-helm@v4 + with: { version: "${{ env.HELM_VERSION }}" } + - name: Create kind cluster + uses: helm/kind-action@v1.10.0 + with: { version: "${{ env.KIND_VERSION }}", node_image: "${{ env.KIND_NODE_IMAGE }}", cluster_name: fed-uninstall } + - name: Pull public federation image + kind load + run: | + docker pull "docker.io/${FED_IMAGE}:${IMAGE_TAG}" + kind load docker-image "docker.io/${FED_IMAGE}:${IMAGE_TAG}" --name fed-uninstall + - name: Install (0-sidecar — license-free, no ES needed) then uninstall + run: | + helm install fed "${CHART_DIR}" + kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=softclient4es-federation --timeout=180s + helm uninstall fed + sleep 5 + leftover=$(kubectl get all,configmap -l app.kubernetes.io/instance=fed --no-headers 2>/dev/null | grep -v '^kubernetes ' | wc -l) + [ "$leftover" = "0" ] || { echo "::error::helm uninstall left $leftover resources"; kubectl get all,configmap -l app.kubernetes.io/instance=fed; exit 1; } diff --git a/softclient4es-federation/README.md b/softclient4es-federation/README.md index 6c7c234..78a8427 100644 --- a/softclient4es-federation/README.md +++ b/softclient4es-federation/README.md @@ -88,6 +88,8 @@ helm uninstall fed | `federation.upgradeUrl` | `https://portal.softclient4es.com/pricing` | `FEDERATION_UPGRADE_URL`. | | `telemetry.enabled` | `true` | `SOFTCLIENT4ES_TELEMETRY_ENABLED` daily-ping opt-out (`false` opts out). | | `license.secretName` | `""` | Secret holding license/API key; empty = Community. | +| `license.publicKeySecretName` | `""` | Secret holding the Ed25519 public JWK for OFFLINE license verification → `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY`; empty = use JWKS fetch. | +| `license.publicKeyKey` | `license-public-key` | Data key within `license.publicKeySecretName`. | | `service.type` | `ClusterIP` | Service type. | | `service.port` | `32020` | `FEDERATION_PORT` (Flight SQL); the only port exposed by the Service. | | `resources` | req `1Gi`/`500m`, lim `2Gi`/`1000m` | Container resource requests/limits. | @@ -104,6 +106,7 @@ helm uninstall fed | `sidecars` | `[]` | Per-ES-version sidecars — see the "Sidecars" section below. | | `sidecarDefaults` | (see `values.yaml`) | Shared resource/probe/security defaults applied to every sidecar. | | `test.image` | `""` (→ `python:3.12-slim`) | Image for the `helm test` smoke Job; pin a pre-baked ADBC image for air-gapped clusters. | +| `test.adbcVersion` | `1.6.0` | ADBC driver version the smoke Job `pip install`s at runtime (when `test.image` is not pre-baked). | ## Sidecars — federating one or more Elasticsearch clusters @@ -182,7 +185,21 @@ exec is the compat path — see the operator guide). With sidecars configured, `helm test fed` runs a Job that connects to the federation Flight SQL endpoint and asserts `GetCatalogs` returns one catalog per sidecar (`len(sidecars)`). The 2-sidecar smoke requires a Pro/Enterprise license (the quota -gate above) and reachable backing ES for each sidecar. +gate above) and reachable backing ES for each sidecar. The Job **retries the connect + +`GetCatalogs` for up to ~60 s** — the federation is NotReady until it has discovered every +downstream (gRPC readiness is all-or-nothing), so a fresh install needs a few seconds. + +The test Pod defaults to `python:3.12-slim` and `pip install`s the ADBC driver at runtime +(needs PyPI reachability). For air-gapped or rate-limited clusters, pin a pre-baked ADBC image +with `--set test.image=` (and `--set test.adbcVersion=` to control the driver version +when the runtime install IS used). This is the same Job CI runs. + +> **Offline license verification (`license.publicKeySecretName`).** When the federation must +> verify a license JWT WITHOUT reaching the license server's JWKS endpoint (air-gapped clusters, +> or a JWT whose `kid` is not in the prod JWKS), set `license.publicKeySecretName` to a Secret +> whose `license.publicKeyKey` data key holds the matching Ed25519 public JWK. It is mounted as +> `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY` (the air-gap path in the license verifier). Leave empty (the +> default) to use the normal JWKS fetch — it renders nothing, so the golden render is unaffected. ## Secrets, TLS & Ingress @@ -317,3 +334,83 @@ the install command, and the `SHOW CATALOGS` smoke expectation: Install any of them with `helm install softclient4es-federation softclient4es-federation -f examples//values.yaml` (create the referenced Secrets first — see each example's README). + +## CI / CD coverage (Story 16.5) + +Every PR touching `softclient4es-federation/**` runs +[`.github/workflows/federation-helm.yml`](../.github/workflows/federation-helm.yml): static +checks (lint + template + `kubeconform -strict` + golden-file diff), an image-availability +prerequisite (pulls the public DockerHub federation + sidecar images), and per-topology / +per-ES-version / secret-backend / upgrade / uninstall installs on ephemeral `kind` clusters. +The smoke test is the chart's own `helm test` Job (ADBC Flight SQL `GetCatalogs`, asserting one +catalog per sidecar) — customers run the exact same `helm test fed` post-install that CI runs. + +This chart repo is **chart-only** (no sbt build): CI never builds images. The live-install jobs +`docker pull` the public DockerHub images and `kind load` them. Until those images are published +(OQ-1), the live-install jobs **probe image availability and skip with a CI annotation** rather +than fail — the static-validation gates run unconditionally on every PR with zero external deps. + +### Tested vs best-effort matrix + +| Scenario | Gate | Tier | License | Notes | +|---|---|---|---|---| +| `helm lint` (chart + each example) | static | **tested** (blocker) | none | unconditional | +| `helm template` + `kubeconform -strict` | static | **tested** (blocker) | none | per example | +| Golden-file diff (template drift) | static | **tested** (blocker) | none | per example + the 16.1/16.2/16.3 goldens | +| Zero `kind: Secret` rendered | static | **tested** (blocker) | none | 16.3 contract | +| `heterogeneous-ready` discriminator | static | **tested** (blocker) | none | greps the `type = "duckdb-attach"` marker (the only signal vs `three-region` — the goldens are byte-identical) | +| Install single-cluster (1×ES8) | kind | **tested** | none (Community) | `GetCatalogs` == 1 | +| Install three-region (2×ES8 + 1×ES9) | kind | **best-effort\*** | **Pro** | mixed-version; `== 3` | +| Install heterogeneous-ready (3×ES) | kind | **best-effort\*** | **Pro** | R2b placeholders inactive; `== 3` | +| Per-ES-version sidecar (6/7/8/9) | kind | **tested** | none | 1 sidecar each; `== 1` | +| Secret backend: raw K8s Secret | kind | **tested** | none | 1 sidecar Secret-backed | +| Secret backend: SealedSecrets | kind | **tested** | none | controller installed + re-sealed in-cluster | +| Secret backend: External Secrets Operator (ESO) | — | **best-effort / documented** | — | needs an external store; examples shipped (16.3), not CI-run | +| Secret backend: Vault Agent Injector | — | **best-effort / documented** | — | needs Vault; documented (16.3) | +| Upgrade single → three-region | kind | **best-effort\*** | **Pro** | topology change asserted (+2 sidecar Deployments) | +| Uninstall clean | kind | **tested** | none | `kubectl get all` empty | + +\* The multi-cluster (Pro) tiers run only when ALL of: (1) the federation image bundles the +JWT-verifying SPI (a **Pro-capable** image — an OSS-only image ships only the Community SPI and +cannot verify ANY Pro JWT), (2) the `SC4ES_PRO_TEST_JWT` repo secret, and (3) the +`SC4ES_TEST_PUBLIC_KEY` repo secret (injected as `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY` via +`license.publicKeySecretName` so the JWT verifies offline) are present. Otherwise they are +**skipped with a CI annotation** (not a failure). A **single-cluster** federation is license-FREE +(Community `maxClusters=1`) and is always tested; the static three-region/heterogeneous golden +proves the mixed-version RENDER on every PR even when the live multi-cluster install is skipped. + +### Image tags in CI + +The live-install jobs resolve the chart's DEFAULT image refs (federation `image.tag:""` → +`appVersion`; sidecar tag → `appVersion`), `docker pull` those public DockerHub tags, and +`kind load` them — no per-image `--set image.tag` override, so the federation + sidecar tags +stay consistent with the committed goldens (which are `appVersion` renders). Until the images +are published (OQ-1), an availability probe (`docker manifest inspect`) gates each live-install +job: if a required image is absent the job is **skipped with a `::warning::` annotation**, never +failed. The golden gate renders with NO `--set image.tag`. + +### CI failure modes (troubleshooting) + +| CI job fails with… | Likely cause | Fix | +|---|---|---| +| golden-file diff non-empty | a template change was not regenerated | run the regen commands above; commit the new golden if the change is intentional | +| `kubeconform` invalid resource | a manifest field renamed/typo (e.g. `replicaCount` vs `replicas`) | fix the template; `replicas` is the Deployment field | +| `heterogeneous-ready` discriminator fails | `three-region` and `heterogeneous-ready` overlays drifted (one copied over the other) | restore the commented R2b `duckdb-attach` preview in `heterogeneous-ready` (the goldens are byte-identical — this grep is the only signal) | +| federation pod CrashLoopBackOff (3 sidecars, license supplied but Community at runtime) | the image lacks the JWT SPI — it cannot verify the Pro JWT → falls back to Community → `maxClusters=1` exceeded | use a Pro-CAPABLE federation image (JWT SPI on classpath, 16.1 OQ-5); injecting a JWT into an OSS-only image does nothing | +| federation pod CrashLoopBackOff (`InvalidLicense: Unknown key ID: …`) | the JWT verification key didn't resolve (no JWKS entry for the kid AND no `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY`) | set `license.publicKeySecretName` → `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY` (the offline verifier path), or ensure the license-server JWKS carries the kid | +| federation pod CrashLoopBackOff (3 sidecars, no license at all) | Community `maxClusters=1` exceeded | supply a Pro/Enterprise license (`license.secretName`) — by design | +| federation pod CrashLoopBackOff (`validate()` / `FlightCredentials`) | Secret-backed cred didn't arrive (wrong key / ESO sync lag) | check the Secret exists + keys match the contract table above; self-heals on next restart | +| federation NotReady, smoke connect-refused | a sidecar's backing ES is down/unreachable (all-or-nothing gRPC readiness) | ensure every sidecar's ES is reachable; or set `federation.probes.useGrpc=false` for partial availability | +| `helm test` count mismatch | a sidecar failed discovery (ES down, or wrong `elasticsearch.url`) | check sidecar + ES logs; verify each sidecar's `elasticsearch.url` resolves in-cluster | +| image pull error (federation OR sidecar) | the public DockerHub tag is not published yet (OQ-1) | the availability probe should skip the live-install job until the image is published; once published, the job `docker pull`s + `kind load`s the public tag | +| live-install job skipped with a `::warning::` | the required public DockerHub image is not yet published (OQ-1) | expected until release; the static-validation gates still run unconditionally | +| ES pod crash (`max virtual memory areas …`) | `vm.max_map_count` too low | `sudo sysctl -w vm.max_map_count=262144` on the runner before ES starts (the workflow does this) | + +### Duration & sharding + +Static checks < 2 min; image pull ~1–2 min; each install job ~6–9 min (when the public images +exist — otherwise the live jobs skip in seconds). With the matrix in parallel, wall-clock ≈ +image-pull + slowest install ≈ ~12 min (< 30 min budget). If runner contention serializes the +matrix past 30 min, move the per-ES-version + secret-backend jobs to a `schedule:` nightly +trigger (uncomment the `schedule:` block in the workflow) and keep PRs to static-checks + +single-cluster + three-region + uninstall. diff --git a/softclient4es-federation/templates/deployment.yaml b/softclient4es-federation/templates/deployment.yaml index 28005e0..a8dfc33 100644 --- a/softclient4es-federation/templates/deployment.yaml +++ b/softclient4es-federation/templates/deployment.yaml @@ -78,6 +78,19 @@ spec: key: {{ .Values.license.apiKeyKey | default "api-key" }} optional: true {{- end }} + {{- /* Story 16.5 (FACT F #2): offline license public-key for air-gapped / + test-signed JWT verification (LicenseKeyVerifier.loadPublicKey Step 3). + Gated INDEPENDENTLY of license.secretName so the multi-sidecar CI can + point it at the same Secret. Renders nothing at the "" default, so the + golden and every example are byte-stable. */}} + {{- if .Values.license.publicKeySecretName }} + - name: SOFTCLIENT4ES_LICENSE_PUBLIC_KEY + valueFrom: + secretKeyRef: + name: {{ .Values.license.publicKeySecretName }} + key: {{ .Values.license.publicKeyKey | default "license-public-key" }} + optional: true + {{- end }} {{- if gt (len .Values.sidecars) 0 }} # Point Typesafe Config at the mounted ConfigMap so ConfigFactory.load() # picks up the `servers` map (which has NO env override). The native-packager diff --git a/softclient4es-federation/templates/tests/smoke-test.yaml b/softclient4es-federation/templates/tests/smoke-test.yaml index 239955c..2157bd9 100644 --- a/softclient4es-federation/templates/tests/smoke-test.yaml +++ b/softclient4es-federation/templates/tests/smoke-test.yaml @@ -12,24 +12,44 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: {{ .Values.test.image | default "python:3.12-slim" | quote }} command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql=={{ .Values.test.adbcVersion | default "1.6.0" }} \ + adbc_driver_manager=={{ .Values.test.adbcVersion | default "1.6.0" }} python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "{{ include "softclient4es-federation.fullname" . }}" - conn = dbapi.connect(f"grpc://{host}:{{ .Values.service.port }}") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:{{ .Values.service.port }}" expected = {{ len .Values.sidecars }} - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY {{- end }} diff --git a/softclient4es-federation/tests/golden/example-heterogeneous-ready.yaml b/softclient4es-federation/tests/golden/example-heterogeneous-ready.yaml index e305683..32e34bf 100644 --- a/softclient4es-federation/tests/golden/example-heterogeneous-ready.yaml +++ b/softclient4es-federation/tests/golden/example-heterogeneous-ready.yaml @@ -926,23 +926,43 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: "python:3.12-slim" command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql==1.6.0 \ + adbc_driver_manager==1.6.0 python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "fed-softclient4es-federation" - conn = dbapi.connect(f"grpc://{host}:32020") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:32020" expected = 3 - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY diff --git a/softclient4es-federation/tests/golden/example-single-cluster.yaml b/softclient4es-federation/tests/golden/example-single-cluster.yaml index dbafbab..a94e4b8 100644 --- a/softclient4es-federation/tests/golden/example-single-cluster.yaml +++ b/softclient4es-federation/tests/golden/example-single-cluster.yaml @@ -399,23 +399,43 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: "python:3.12-slim" command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql==1.6.0 \ + adbc_driver_manager==1.6.0 python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "fed-softclient4es-federation" - conn = dbapi.connect(f"grpc://{host}:32020") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:32020" expected = 1 - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY diff --git a/softclient4es-federation/tests/golden/example-three-region.yaml b/softclient4es-federation/tests/golden/example-three-region.yaml index e305683..32e34bf 100644 --- a/softclient4es-federation/tests/golden/example-three-region.yaml +++ b/softclient4es-federation/tests/golden/example-three-region.yaml @@ -926,23 +926,43 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: "python:3.12-slim" command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql==1.6.0 \ + adbc_driver_manager==1.6.0 python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "fed-softclient4es-federation" - conn = dbapi.connect(f"grpc://{host}:32020") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:32020" expected = 3 - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY diff --git a/softclient4es-federation/tests/golden/ingress-tls.yaml b/softclient4es-federation/tests/golden/ingress-tls.yaml index d86c77d..c124858 100644 --- a/softclient4es-federation/tests/golden/ingress-tls.yaml +++ b/softclient4es-federation/tests/golden/ingress-tls.yaml @@ -406,23 +406,43 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: "python:3.12-slim" command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql==1.6.0 \ + adbc_driver_manager==1.6.0 python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "fed-softclient4es-federation" - conn = dbapi.connect(f"grpc://{host}:32020") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:32020" expected = 1 - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY diff --git a/softclient4es-federation/tests/golden/secret-auth.yaml b/softclient4es-federation/tests/golden/secret-auth.yaml index 20ca27a..bc47a30 100644 --- a/softclient4es-federation/tests/golden/secret-auth.yaml +++ b/softclient4es-federation/tests/golden/secret-auth.yaml @@ -438,23 +438,43 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: "python:3.12-slim" command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql==1.6.0 \ + adbc_driver_manager==1.6.0 python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "fed-softclient4es-federation" - conn = dbapi.connect(f"grpc://{host}:32020") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:32020" expected = 1 - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY diff --git a/softclient4es-federation/tests/golden/two-sidecars.yaml b/softclient4es-federation/tests/golden/two-sidecars.yaml index bf1726d..8353c81 100644 --- a/softclient4es-federation/tests/golden/two-sidecars.yaml +++ b/softclient4es-federation/tests/golden/two-sidecars.yaml @@ -612,23 +612,43 @@ spec: restartPolicy: Never containers: - name: show-catalogs + # Default: a public python image; CI + air-gapped clusters MUST pin `test.image` + # to a PRE-BAKED image with adbc_driver_flightsql installed (no runtime pip). image: "python:3.12-slim" command: ["sh", "-c"] args: - | - pip install --quiet adbc_driver_flightsql==1.6.0 adbc_driver_manager==1.6.0 + set -e + # If the image is NOT pre-baked, install ADBC at runtime (needs PyPI reachability). + python -c "import adbc_driver_flightsql" 2>/dev/null || \ + pip install --quiet adbc_driver_flightsql==1.6.0 \ + adbc_driver_manager==1.6.0 python - <<'PY' - import sys + import sys, time import adbc_driver_flightsql.dbapi as dbapi host = "fed-softclient4es-federation" - conn = dbapi.connect(f"grpc://{host}:32020") - # GetCatalogs: one catalog per registered sidecar alias. - cats = [c["catalog_name"] for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] - cats = [c for c in cats if c] # drop empty/system catalog if present + uri = f"grpc://{host}:32020" expected = 2 - print("catalogs:", cats) - if len(cats) != expected: - print(f"FAIL: expected {expected} catalogs, got {len(cats)}", file=sys.stderr) - sys.exit(1) - print("OK") + # The federation only reports its catalogs once it has DISCOVERED every downstream + # (gRPC readiness is all-or-nothing). Retry the connect + GetCatalogs for up to ~60s. + deadline = time.time() + 60 + last_err = None + while time.time() < deadline: + try: + conn = dbapi.connect(uri) + # GetCatalogs: one catalog_name row per registered sidecar alias (VERIFIED + # FederationFlightProducer.getStreamCatalogs — needs NO default downstream). + cats = [c["catalog_name"] + for c in conn.adbc_get_objects(depth="catalogs").read_all().to_pylist()] + cats = [c for c in cats if c] # drop empty/system catalog if present + conn.close() + if len(cats) == expected: + print(f"OK: {len(cats)} catalogs: {cats}") + sys.exit(0) + last_err = f"expected {expected} catalogs, got {len(cats)}: {cats}" + except Exception as e: # connect refused while federation still NotReady + last_err = f"connect/get_objects error: {e}" + time.sleep(3) + print(f"FAIL: {last_err}", file=sys.stderr) + sys.exit(1) PY diff --git a/softclient4es-federation/values.yaml b/softclient4es-federation/values.yaml index fc640bc..07af596 100644 --- a/softclient4es-federation/values.yaml +++ b/softclient4es-federation/values.yaml @@ -115,6 +115,16 @@ license: secretName: "" licenseKeyKey: license-key # Secret data key holding the JWT license -> SOFTCLIENT4ES_LICENSE_KEY apiKeyKey: api-key # Secret data key holding the API key -> SOFTCLIENT4ES_API_KEY + # ── Offline license public-key (Story 16.5, FACT F #2) ──────────────────────── + # OPTIONAL. When the federation must verify a license JWT WITHOUT reaching the + # license server's JWKS endpoint (air-gapped clusters, or a test-signed JWT whose + # `kid` is not in the prod JWKS), point publicKeySecretName at a Secret whose + # `publicKeyKey` data key holds the matching Ed25519 public JWK. It is mounted as + # SOFTCLIENT4ES_LICENSE_PUBLIC_KEY — the air-gap path in LicenseKeyVerifier.loadPublicKey + # (Step 3, VERIFIED). Leave empty (the default) to use the normal JWKS fetch. Renders + # NOTHING when empty, so the golden render and every shipped example are unaffected. + publicKeySecretName: "" + publicKeyKey: license-public-key # Secret data key holding the Ed25519 public JWK -> SOFTCLIENT4ES_LICENSE_PUBLIC_KEY service: type: ClusterIP @@ -313,9 +323,12 @@ sidecarDefaults: drop: - ALL -# `helm test` smoke Job (Story 16.2). Connects to the federation Flight SQL endpoint -# and asserts GetCatalogs (one catalog per registered sidecar) == len(sidecars). +# `helm test` smoke Job (Story 16.2, hardened for CI in Story 16.5). Connects to the +# federation Flight SQL endpoint and asserts GetCatalogs (one catalog per registered +# sidecar) == len(sidecars), with a bounded connect-retry (the federation is NotReady +# until ALL downstreams are discovered — the gRPC readiness aggregate is all-or-nothing). # The default image pip-installs ADBC at runtime (needs PyPI reachability) — pin an -# internal/pre-baked image for air-gapped or rate-limited clusters. +# internal/pre-baked image (test.image) for air-gapped or rate-limited clusters. test: image: "" # default "python:3.12-slim" when empty + adbcVersion: "1.6.0" # adbc_driver_flightsql / adbc_driver_manager version (runtime pip when image not pre-baked) From 6b046e06f677a58e444d0da90a9150bc8bf7da81 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Manciot?= Date: Tue, 23 Jun 2026 10:48:48 +0200 Subject: [PATCH 3/3] =?UTF-8?q?Story=2016.6=20=E2=80=94=20Operator=20guide?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the canonical operator guide (softclient4es-federation/docs/operator-guide.md) that ships with the chart: prerequisites, single-cluster + multi-cluster paths, configuration reference, secret-backend chooser, licensing quota, migration, upgrades/rollback, troubleshooting + SRE incident-triage walkthrough. Authoritative copy — mirror language and private-repo references removed; working-directory convention uses softclient4es-federation/. Same Chart.yaml/README/CI rewrites as prior stories (softclient4es.dev metadata, public DockerHub images, chart-only CI). Closes #12 Refs #11 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../docs/operator-guide.md | 225 ++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 softclient4es-federation/docs/operator-guide.md diff --git a/softclient4es-federation/docs/operator-guide.md b/softclient4es-federation/docs/operator-guide.md new file mode 100644 index 0000000..35e9d91 --- /dev/null +++ b/softclient4es-federation/docs/operator-guide.md @@ -0,0 +1,225 @@ +# SoftClient4ES Federation — Operator Guide + +> **Authoritative guide.** This is the canonical operator guide that ships with the federation Helm chart at `softclient4es-federation/docs/operator-guide.md`. The web and core-docs copies point here. + +The SoftClient4ES federation Helm chart deploys a **cross-cluster Arrow Flight SQL coordinator** (the *federation server*) plus one **per-ES-version sidecar** for each Elasticsearch cluster you want to federate. One `values.yaml` + `helm install` replaces 30+ minutes of hand-written HOCON, Kubernetes manifests, Secrets, and probe wiring. + +## 1. Prerequisites + +- A Kubernetes cluster (v1.27+ recommended for native gRPC readiness probes; on **older clusters set `federation.probes.useGrpc: false`** for TCP readiness — no extra binary needed; see §9/§10). +- **Helm 3**. +- One or more Elasticsearch clusters **reachable from the K8s cluster** (each sidecar opens a connection to exactly one ES cluster of its compiled-against major version). +- A container runtime able to pull from **public DockerHub** (`docker.io/softnetwork/...`). For rate-limit avoidance on first install, configure `imagePullSecrets` with an authenticated DockerHub account. +- (Multi-cluster only) a **Pro or Enterprise license** — see §7. Single-cluster federation runs free on the Community tier. + +> **Image availability:** the federation image `docker.io/softnetwork/softclient4es-federation` is published to public DockerHub at R1 release. The four sidecar images `docker.io/softnetwork/softclient4es{6,7,8,9}-arrow-flight-sql` are already published. Always pin `image.tag` to the published release tag in your `values.yaml`. + +**Working directory convention.** Every `cp examples/…` and `helm install … ./softclient4es-federation` command in this guide is run **from the chart directory `softclient4es-federation/`** (where `Chart.yaml`, `examples/`, and the `./softclient4es-federation` chart path resolve). `cd` there first: + +```bash +cd softclient4es-federation/ # all commands below run from here +``` + +The release name you pass to `helm install …` (this guide uses `fed`) becomes part of the in-cluster Service DNS — `-softclient4es-federation..svc.cluster.local`. Keep it consistent across `install`, `upgrade`, `test`, and `rollback`. + +## 2. The 5-minute path — single cluster + +```bash +# 1. Copy the starter example +cp examples/single-cluster/values.yaml my-values.yaml + +# 2. Edit my-values.yaml — set the one sidecar's ES coordinates + version: +# sidecars[0].elasticsearchVersion: 8 +# sidecars[0].elasticsearch.url: https://es.mycorp.internal:9200 +# sidecars[0].elasticsearch.credentialsSecretName: my-es-secret # see §6 +# image.tag: + +# 3. Create your ES credentials Secret (raw example; §6 for SealedSecrets/ESO/Vault) +kubectl create secret generic my-es-secret \ + --from-literal=es-auth-method=basic \ + --from-literal=es-username=elastic \ + --from-literal=es-password='' + +# 4. Install +helm install fed ./softclient4es-federation -f my-values.yaml + +# 5. Validate +helm test fed # runs GetCatalogs; passes when the federation lists 1 catalog +``` + +Single-cluster is **license-free** (Community `maxClusters=1`). The federation exposes Flight SQL on `ClusterIP` port **32020**; reach it in-cluster at `fed-softclient4es-federation..svc.cluster.local:32020`, or expose it via an Ingress (§6, TLS). + +See [`examples/single-cluster/README.md`](../examples/single-cluster/README.md) for the full single-cluster narrative + topology diagram. + +## 3. Multi-cluster — three regions, mixed ES versions + +```bash +cp examples/three-region/values.yaml my-values.yaml +# Per-region sidecars (us-east-1 ES8 default · eu-west-1 ES8 · ap-south-1 ES9), +# each with its own elasticsearch.credentialsSecretName + auth.credentialsSecretName. + +# 1. Create the Pro/Enterprise license Secret (REQUIRED — 3 clusters > Community's quota of 1; see §7). +# Replace with the JWT from the portal. +kubectl create secret generic sc4es-pro-license \ + --from-literal=license-key='' + +# 2. Create the per-region ES + sidecar-auth Secrets the example references +# (one ES Secret + one sidecar-auth Secret per region; see §6 for SealedSecrets/ESO/Vault). +# Replace every <…> with your real coordinates. +for r in us-east-1 eu-west-1 ap-south-1; do + kubectl create secret generic sc4es-es-$r \ + --from-literal=es-auth-method=basic \ + --from-literal=es-username='' \ + --from-literal=es-password='' + kubectl create secret generic sc4es-arrow-$r \ + --from-literal=arrow-bearer-token='' +done + +# 3. Install (REQUIRES the Pro/Enterprise license Secret above — without it the federation CrashLoops, see §7/§10). +helm install fed ./softclient4es-federation -f my-values.yaml +helm test fed # GetCatalogs → 3 catalogs +``` + +> The example's `values.yaml` references these Secrets by name (`license.secretName: sc4es-pro-license`, each region's `elasticsearch.credentialsSecretName: sc4es-es-` + `auth.credentialsSecretName: sc4es-arrow-`). **Create them BEFORE `helm install`** — a missing Secret is the #1 first-install failure (a loud, self-healing CrashLoop; §6/§10). The exact key names (`license-key`, `es-auth-method`/`es-username`/`es-password`, `arrow-bearer-token`) are the chart's expected Secret data keys (§6). See [`examples/three-region/README.md`](../examples/three-region/README.md) for the full per-region narrative (GDPR data-residency, SRE incident-triage, mixed-version migration). + +Each `sidecars[]` entry deploys one Deployment + one Service and registers one `arrow.flight.federation.servers.` entry (host = `..svc.cluster.local:32010`). Mixed ES versions are first-class — the chart selects `softclient4es{6,7,8,9}-arrow-flight-sql` per `elasticsearchVersion`. **Adding a cluster** = add a `sidecars[]` entry + `helm upgrade`; **removing** = delete the entry + `helm upgrade`. + +> ⚠️ **All-or-nothing readiness (SPOF).** With the default `federation.probes.useGrpc: true`, the federation is Ready only if **every** sidecar's backing ES is reachable — one down region pulls the whole federation out of its Service. For partial availability set `federation.probes.useGrpc: false` (TCP readiness on 32020); the federation then stays Ready and fails only the queries that touch the down region. See §9. + +## 4. The R2b-ready path + +```bash +cp examples/heterogeneous-ready/values.yaml my-values.yaml +``` + +This is the three-region topology **plus commented-out R2b `duckdb-attach` placeholders** (PostgreSQL / MySQL / Snowflake) that show how heterogeneous sources slot into the same `servers` map when R2b ships (Epic 25/26). **In R1 they stay commented** — `GetCatalogs` returns 3, not 6. R1 has no `values.yaml` key for `duckdb-attach` (the `servers` map is ConfigMap-rendered from `sidecars[]` Flight SQL servers only). **Uncommenting them later counts toward the license quota** (`clusterCount` includes attach servers) — 3 ES + 3 attach = 6 > Pro's 5 → Enterprise. + +See [`examples/heterogeneous-ready/README.md`](../examples/heterogeneous-ready/README.md) for the R2b preview narrative. + +## 5. Configuration reference + +> `values.yaml` field → HOCON path / env var. **Verify against the chart's own `values.yaml` and README; this table is the authoritative mapping.** + +**Federation** (image `softclient4es-federation`; HOCON root `arrow.flight.federation.*`): + +| `values.yaml` | HOCON | Env var | Default | +|---|---|---|---| +| `replicaCount` | — | — | 1 | +| `image.repository` / `image.tag` / `image.pullPolicy` | — | — | `docker.io/softnetwork/softclient4es-federation` / `appVersion` / `IfNotPresent` | +| `federation.maxMemory` | `max-memory` | `FEDERATION_MAX_MEMORY` | `512m` | +| `federation.queryTimeoutSeconds` | `query-timeout-seconds` | `FEDERATION_QUERY_TIMEOUT` | 30 | +| `federation.health.port` | `health.port` | `FEDERATION_HEALTH_PORT` | 32021 | +| `federation.health.probeTimeoutSeconds` | `health.probe-timeout-seconds` | `FEDERATION_HEALTH_PROBE_TIMEOUT` | 5 | +| `federation.duckdb.path` | `duckdb.path` | `FEDERATION_DUCKDB_PATH` | `:memory:` | +| `federation.upgradeUrl` | `upgrade-url` | `FEDERATION_UPGRADE_URL` | portal pricing URL | +| (`host`/`port` fixed by chart) | `host` / `port` | `FEDERATION_HOST` / `FEDERATION_PORT` | `0.0.0.0` / 32020 | +| `federation.probes.useGrpc` | — (probe wiring) | — | true | +| `federation.tls.{enabled,secretName}` | — (Ingress) | — | false | +| `ingress.{enabled,className,annotations,hosts,tls}` | — (Ingress) | — | false | +| `license.publicKeySecretName` (+ `license.publicKeyKey`) | — | `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY` | `""` / `license-public-key` | + +**Sidecar** (per `sidecars[]`; image auto-selected by `elasticsearchVersion`; HOCON root `arrow.flight.*` + elasticsql core `elastic.credentials.*`): + +| `values.yaml` | HOCON / config | Env var | Default | +|---|---|---|---| +| `sidecars[].elasticsearchVersion` (`6\|7\|8\|9`) | — (image selection) | — | — | +| `sidecars[].name` | `servers.` key (RFC1123, unique) | — | — | +| `sidecars[].elasticsearch.url` (or `.scheme/.host/.port`) | `elastic.credentials.*` | `ELASTIC_SCHEME` / `ELASTIC_HOST` / `ELASTIC_PORT` | — (NO single ES-URL env) | +| `sidecars[].elasticsearch.credentialsSecretName` | `elastic.credentials.*` | `ELASTIC_AUTH_METHOD` / `ELASTIC_CREDENTIALS_{USERNAME,PASSWORD,API_KEY,BEARER_TOKEN}` | — | +| `sidecars[].arrow.batchSize` | `batch-size` | `ARROW_BATCH_SIZE` | 1000 | +| `sidecars[].arrow.queryTimeoutSeconds` | `query-timeout-seconds` | `ARROW_QUERY_TIMEOUT_SECONDS` | 120 | +| `sidecars[].arrow.maxMemory` | `join.max-memory` | `ARROW_JOIN_MAX_MEMORY` | `256m` | +| `sidecars[].auth.{method,credentialsSecretName}` | `auth.*` + federation `servers..credentials` | `ARROW_AUTH_{METHOD,USERNAME,PASSWORD,BEARER_TOKEN,API_KEY}` + `CONFIG_FORCE_*` | `none` | +| `sidecars[].default` | `servers..default` | — | false (≤1 true) | +| `sidecars[].alias` | `servers..alias` | — | `name` | +| `sidecars[].tls` | `servers..tls` (outgoing) | — | false | +| `sidecars[].replicaCount` / `.resources` / `.image.{repository,tag}` | — | — | 1 / — / version-default | + +**License & telemetry** (elasticsql `licensing`; HOCON root `softclient4es.*`): + +| `values.yaml` | HOCON | Env var | Default | +|---|---|---|---| +| `license.secretName` → `license-key` | `softclient4es.license.key` | `SOFTCLIENT4ES_LICENSE_KEY` | `""` (Community) | +| `license.secretName` → `api-key` | `softclient4es.license.api-key` | `SOFTCLIENT4ES_API_KEY` | `""` | +| `telemetry.enabled` | `softclient4es.telemetry.enabled` | `SOFTCLIENT4ES_TELEMETRY_ENABLED` | true | + +> **No env override** (NOT chart-exposed): `softclient4es.license.{connect-timeout,read-timeout,grace-period,cache-dir,refresh.enabled,refresh.interval}` and `softclient4es.license.telemetry.enabled` (the license-refresh-metrics switch — DISTINCT from the daily-ping `softclient4es.telemetry.enabled` above; do not confuse them). + +## 6. Secret management — choosing a backend + +The chart **never creates a `Secret`** — you create it (raw, SealedSecrets, External Secrets Operator, or Vault Agent Injector) and reference it by name. One Secret feeds **both** the sidecar (`ARROW_AUTH_*`, `ELASTIC_*`) and, for federation→sidecar auth, the federation (`CONFIG_FORCE_*` via Typesafe Config `override_with_env_vars`). The four backends are documented in depth in [`docs/secret-backends.md`](secret-backends.md), with ready-to-adapt manifests under [`examples/sealed-secrets/`](../examples/sealed-secrets/) and [`examples/external-secrets/`](../examples/external-secrets/). In short: + +- **Raw `kubectl create secret`** — simplest; fine for dev / GitOps with encrypted-at-rest etcd. Keys: `es-auth-method`/`es-username`/`es-password`/`es-api-key`/`es-bearer-token` (ES), `arrow-username`/`arrow-password`/`arrow-bearer-token`/`arrow-api-key` (sidecar auth), `license-key`/`api-key` (license), `tls.crt`/`tls.key` (TLS). +- **SealedSecrets** — commit an encrypted `SealedSecret` to Git; the in-cluster controller decrypts it. Re-seal per controller (certs are per-controller; `kubeseal` CLI version must match the controller `appVersion`, not the Helm chart version). +- **External Secrets Operator (ESO)** — sync from AWS/GCP Secret Manager or Vault. Examples use `external-secrets.io/v1` (the stable API; `v1beta1` was removed at ESO v0.17). +- **Vault Agent Injector** — inject secrets as files/env via pod annotations. + +> **RFC1123 sidecar names are mandatory.** The federation→sidecar credential injection mangles the sidecar `name` into a `CONFIG_FORCE_*` env path; a name with `_`/`.`/uppercase mangles wrong and the federation `sys.exit(1)`s. Use `[a-z0-9-]` names. **A wrong/missing Secret is a loud, SELF-HEALING CrashLoop** — the pod restarts and boots Ready once the Secret materializes (e.g. after ESO syncs). Do NOT uninstall; check the Secret's data keys. + +**TLS at the Ingress, not the pod.** The federation listener is plaintext-only (`Location.forGrpcInsecure`). `federation.tls.{enabled,secretName}` drives an **Ingress** `tls:` block (cert-manager `kubernetes.io/tls`); since Flight SQL is gRPC, the Ingress controller must proxy gRPC — nginx annotation `nginx.ingress.kubernetes.io/backend-protocol: "GRPC"` (NOT `GRPCS`; the pod upstream is plaintext h2c). This is DISTINCT from `sidecars[].tls` (outgoing per-downstream client TLS → `servers..tls`). In-cluster plaintext clients hit the Service (32020) directly. + +## 7. Licensing + +The federation reads its license from a referenced Secret as `SOFTCLIENT4ES_LICENSE_KEY` (offline Ed25519 JWT) and/or `SOFTCLIENT4ES_API_KEY` (automated portal provisioning). The gate is a **per-platform `maxClusters` quota**, enforced at startup against the number of `servers` (sidecars): + +| Tier | `maxClusters` | Federation clusters you can run | +|---|---|---| +| **Community** (no license) | 1 | single-cluster federation — **free** | +| **Pro** | 5 | up to 5 sidecars / clusters | +| **Enterprise** | unlimited | any | + +Exceeding the quota → the federation logs the over-quota error and `sys.exit(1)` → **CrashLoopBackOff by design** (see §10). Federation is NOT a paid feature — single-cluster is the free adoption tier; the quota is on *cluster count*. To opt out of the daily anonymous usage ping, set `telemetry.enabled: false` (→ `SOFTCLIENT4ES_TELEMETRY_ENABLED=false`); this has zero impact on functionality or your license. + +> **Offline verification.** For air-gapped or strict-egress clusters, mount the Ed25519 public JWK via `license.publicKeySecretName` (key `license.publicKeyKey`, default `license-public-key`) → `SOFTCLIENT4ES_LICENSE_PUBLIC_KEY`. This lets the federation verify a Pro/Enterprise JWT entirely offline (no portal round-trip). It is gated independently of `license.secretName`. + +## 8. ES-version mixing & migration + +Sidecars are independent — there is **no constraint** on mixing ES 6/7/8/9 in one federation at R1. The canonical migration story: run ES 8 sidecars in two regions while you migrate one region to ES 9; flip that region's `sidecars[].elasticsearchVersion` from `8` to `9` and `helm upgrade` once the ES cluster is upgraded. Cross-version JOINs with version-specific SQL features are the one caveat — flag them in testing (advanced cross-version cases surface in customer testing). + +## 9. Upgrades & rollback + +- **Upgrade:** edit `values.yaml`, `helm upgrade fed ./softclient4es-federation -f my-values.yaml`. Adding/removing a sidecar re-renders the federation ConfigMap; the federation pod restarts to pick up the new `servers` map. +- **Rollback:** `helm rollback fed ` (`helm history fed` to list). +- **Secret rotation** is start-time: after changing a Secret, `kubectl rollout restart deployment` for BOTH the sidecar AND the federation (one Secret feeds both — restart both or the two sides drift). +- **Probes:** native gRPC readiness needs K8s 1.27+. On clusters **older than 1.27**, set `federation.probes.useGrpc: false` (TCP readiness on 32020) — this is the recommended fallback and needs **no extra binary**. The `exec`+`grpc_health_probe` approach is NOT a drop-in: the `grpc_health_probe` binary is **not bundled in the federation image**, so you would have to bake it into a custom image (or an init-container copy) yourself before an `exec` probe can run it. For nearly all operators, `useGrpc: false` is the simpler and supported path below 1.27. + +## 10. Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| Federation pod `CrashLoopBackOff`, log "exceeds maxClusters" / `sys.exit(1)` | More sidecars than the license tier allows (Community=1, Pro=5) | Add/upgrade the license Secret (Pro/Enterprise), or reduce `sidecars[]`. §7 | +| `CrashLoopBackOff` after uncommenting R2b `duckdb-attach` | `clusterCount` counts attach servers too → over Pro's 5 | Enterprise license, or keep R2b commented in R1. §4 | +| `CrashLoopBackOff`, "invalid credentials" at config load | `auth.method` ≠ none with creds only in a Secret but `federation.credentialsFromEnv` off, OR a non-RFC1123 sidecar name mangled the `CONFIG_FORCE_*` path | Enable `credentialsFromEnv`; rename sidecar to `[a-z0-9-]`. §6 | +| `CrashLoopBackOff` immediately at boot, no clear error | `readOnlyRootFilesystem` with no writable `/tmp` — the DuckDB JNI native lib can't extract | Ensure the `/tmp` `emptyDir` is present (chart default); don't override the volume. | +| Federation NotReady, "could not connect to sidecar X" | Sidecar pod / its backing ES unreachable; all-or-nothing gRPC readiness pulls the whole federation | Check sidecar pod + ES; for partial availability set `federation.probes.useGrpc: false`. §3/§9 | +| Wrong Secret keys / ESO sync lag | `optional:true` env can't validate Secret CONTENTS → boot-time `validate()` CrashLoop | Fix the Secret's data keys; the pod self-heals on the next restart — do NOT uninstall. §6 | +| License validation failure (**Pro/Enterprise only**) | A license JWT WAS supplied but is malformed / clock-skewed / expired beyond the 14-day grace | Re-provision via the portal (`SOFTCLIENT4ES_API_KEY`); check the JWT and node clock. **N.B.** with NO license (blank `SOFTCLIENT4ES_LICENSE_KEY`) the federation does NOT error — it runs on Community (1-cluster quota); that fallback is expected, not a failure. Community has no API-key/refresh path, so the portal re-provision step applies ONLY to a paid license. | +| TLS cert renewal failure | Cert lives at the **Ingress** (federation pod is plaintext) | Renew the cert-manager Certificate / Ingress `tls` Secret; nginx needs `backend-protocol: "GRPC"`. §6 (TLS) | +| ES connection failure from a sidecar | Wrong `ELASTIC_SCHEME/HOST/PORT/AUTH_METHOD` or ES creds | Verify the ES Secret + reachability; remember there is NO single ES-URL — scheme/host/port are separate. §5 | +| ES-version mismatch | `elasticsearchVersion` doesn't match the backing ES major | Set the sidecar's `elasticsearchVersion` to the ES cluster's major. §8 | + +### SRE incident-triage walkthrough (three-region) + +1. **Alert:** dashboards/queries failing across regions. `kubectl get pods` → the `fed-softclient4es-federation` pod is `0/1 NotReady`. +2. **Confirm the SPOF:** `kubectl describe pod` shows the gRPC readiness probe failing. With `useGrpc:true`, one unreachable region makes the *whole* federation NotReady (all-or-nothing aggregate). +3. **Find the bad region:** `kubectl get pods -l app.kubernetes.io/component=sidecar` → identify the `NotReady` sidecar; `kubectl logs` it for the ES connection error. +4. **Mitigate fast:** re-apply your existing values file AND flip the probe — `helm upgrade fed ./softclient4es-federation -f my-values.yaml --set federation.probes.useGrpc=false` → the federation goes Ready and serves the two healthy regions; only queries touching the down region fail. (Per-alias readiness is an R1.x backlog item.) ⚠️ **Always include `-f my-values.yaml`**: a bare `helm upgrade … --set …` resets every other value to the chart default (Helm does NOT reuse the previous release's values unless you pass `-f` or `--reuse-values`), which would silently revert your three-region `sidecars[]` / license / topology mid-incident. +5. **Fix root cause:** restore the down region's ES / sidecar, then flip `useGrpc` back to `true` for fail-closed semantics. + +## 11. Performance tuning *(placeholder — iterate after R1 telemetry)* + +> Hard numbers land after R1 telemetry. The qualitative levers today: + +- **Scale federation replicas** (`replicaCount`) for concurrent-query throughput; the federation is stateless per query (DuckDB `:memory:`). +- **Scale sidecars** (`sidecars[].replicaCount`, document HA at 3) for per-cluster availability and parallelism. +- **`federation.maxMemory` / `sidecars[].arrow.maxMemory`** size the DuckDB / JOIN engine; raise for large cross-cluster JOINs, watch pod memory limits. +- **`batchSize`** trades latency vs throughput on streaming. +- Resource recommendations per cluster-count tier: TBD — seeded after R1 telemetry. + +## 12. SLA implications + +Pro includes 48h support. This guide is intended to be authoritative enough that the common scenarios (install, add/remove a cluster, mixed-version migration, the over-quota and connectivity failure modes) are self-serviceable without a ticket. + +## For Epic 17 + +Epic 17 (Documentation & Marketing) consumes, for R1 launch-day docs: **this operator guide** (linked/included from the launch docs), the **`examples/` topology folders** ([single-cluster](../examples/single-cluster/README.md), [three-region](../examples/three-region/README.md), [heterogeneous-ready](../examples/heterogeneous-ready/README.md) — each with README + ASCII diagram), and the **SRE incident-triage walkthrough** (§10, linked from the R1 marketing SRE-wedge story).