diff --git a/.github/release-notes-install.md b/.github/release-notes-install.md index 0e23b697..b8245233 100644 --- a/.github/release-notes-install.md +++ b/.github/release-notes-install.md @@ -4,7 +4,7 @@ ${IMAGE_REPO}:${VERSION} ``` -Multi-arch (`linux/amd64`, `linux/arm64`), distroless nonroot base. +Multi-arch (`linux/amd64`, `linux/arm64`), Debian bookworm-slim nonroot base (includes `git` and `openssh-client` for `spec.git.engine: cli`). OCI attestations (SBOM + SLSA provenance) are attached in GHCR and on the repository [Attestations](https://github.com/${GITHUB_REPOSITORY}/attestations) page. Verify the signature: diff --git a/CHANGELOG.md b/CHANGELOG.md index 8e46533b..95edf7a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,9 +27,13 @@ anchor on an older commit. ### Bug Fixes -- **e2e:** Validate collection via inventory HTTP [cd77870](https://github.com/konih/kollect/commit/cd778701acdd95cb4d2bf74e677499b856bbc702) +- **e2e:** Drop legacy sinkRefs from multitenant scope fixture -- **e2e,test:** Stabilize smoke bootstrap and debounce IT [99d72bb](https://github.com/konih/kollect/commit/99d72bb35ba1cffe4a8c7dbd6777bc5ae37665e2) +- **e2e:** Drop removed inventory sinkRefs field [48af9c4](https://github.com/konih/kollect/commit/48af9c4924c150bea36694dac1f691f3b7a88d82) + +- **e2e:** Validate collection via inventory HTTP [ca816a0](https://github.com/konih/kollect/commit/ca816a072b89326d0bc6c6a6dae5f4d2f0ec8159) + +- **e2e,test:** Stabilize smoke bootstrap and debounce IT [3677bb5](https://github.com/konih/kollect/commit/3677bb5dc2b1685def8a7aec05f066604e18985c) - **samples:** Drop legacy sinkRefs from team-a scope [1648ab3](https://github.com/konih/kollect/commit/1648ab3ce0b67e58d8129da909487db91bf3c23b) @@ -40,7 +44,7 @@ anchor on an older commit. ### Features -- **git:** Port transport retry and SSH host keys [b8bab57](https://github.com/konih/kollect/commit/b8bab57eba8ed6f554edad951ee761dc8c39a5a4) +- **git:** Port transport retry and SSH host keys [9f5b17f](https://github.com/konih/kollect/commit/9f5b17f40d0eda7b8e482e2c03cb69aac66f370a) - **controller:** Wire family sink reconcilers and export [8162345](https://github.com/konih/kollect/commit/8162345b399d75ae315d05e06bb25a43c7eb4e0f) diff --git a/Dockerfile b/Dockerfile index 9c21bfc5..da4ff7df 100644 --- a/Dockerfile +++ b/Dockerfile @@ -21,11 +21,18 @@ COPY . . # by leaving it empty we can ensure that the container and binary shipped on it will have the same platform. RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -ldflags="-w -s" -a -o manager cmd/main.go -# Use distroless as minimal base image to package the manager binary -# Refer to https://github.com/GoogleContainerTools/distroless for more details -FROM gcr.io/distroless/static:nonroot@sha256:963fa6c544fe5ce420f1f54fb88b6fb01479f054c8056d0f74cc2c6000df5240 +# Runtime image: Debian slim with git + openssh-client for spec.git.engine=cli and git ls-remote probes. +# go-git export (default engine) does not require the git binary; the CLI path and connection probes do. +FROM debian:bookworm-slim@sha256:0104b334637a5f19aa9c983a91b54c89887c0984081f2068983107a6f6c21eeb + +RUN apt-get update && \ + apt-get install -y --no-install-recommends ca-certificates git openssh-client && \ + rm -rf /var/lib/apt/lists/* && \ + groupadd --gid 65532 nonroot && \ + useradd --uid 65532 --gid 65532 --home-dir /home/nonroot --shell /usr/sbin/nologin --no-create-home nonroot + WORKDIR / -COPY --from=builder /workspace/manager . +COPY --from=builder /workspace/manager /manager USER 65532:65532 ENTRYPOINT ["/manager"] diff --git a/SECURITY.md b/SECURITY.md index d407f6ad..fe647b15 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -35,7 +35,7 @@ Risks to consider when deploying: - Sink endpoints must use verified TLS; credentials must not appear in CR specs or logs. - Restrict egress with `NetworkPolicy` in production. -See [engineering guidelines](docs/development/guidelines.md) for hardening baselines (distroless image, non-root, secret +See [engineering guidelines](docs/development/guidelines.md) for hardening baselines (non-root runtime image, secret handling, supply-chain checks in CI). ## Supply chain (releases) diff --git a/charts/kollect/README.md b/charts/kollect/README.md index f88de26f..c8a84fb5 100644 --- a/charts/kollect/README.md +++ b/charts/kollect/README.md @@ -10,6 +10,13 @@ Installs the [Kollect](https://github.com/konih/kollect) operator and CRDs. helm install kollect ./charts/kollect -n kollect-system --create-namespace ``` +### Container image + +The default image is **Debian bookworm-slim** (UID/GID 65532) with `git` and `openssh-client` so +`spec.git.engine: cli` and `git ls-remote` connection probes work out of the box. Default +`spec.git.engine: go-git` does not require those binaries. Pod `securityContext` defaults are +unchanged (`readOnlyRootFilesystem: true`, capabilities dropped, `/tmp` `emptyDir` for git workdirs). + ## Values | Key | Type | Default | Description | diff --git a/charts/kollect/README.md.gotmpl b/charts/kollect/README.md.gotmpl index 9b54d7d3..ec7940d1 100644 --- a/charts/kollect/README.md.gotmpl +++ b/charts/kollect/README.md.gotmpl @@ -10,6 +10,13 @@ Installs the [Kollect](https://github.com/konih/kollect) operator and CRDs. helm install kollect ./charts/kollect -n kollect-system --create-namespace ``` +### Container image + +The default image is **Debian bookworm-slim** (UID/GID 65532) with `git` and `openssh-client` so +`spec.git.engine: cli` and `git ls-remote` connection probes work out of the box. Default +`spec.git.engine: go-git` does not require those binaries. Pod `securityContext` defaults are +unchanged (`readOnlyRootFilesystem: true`, capabilities dropped, `/tmp` `emptyDir` for git workdirs). + {{ template "chart.valuesSection" . }} Export debouncing is configured per **`KollectInventory.spec.exportMinInterval`** (CRD default diff --git a/charts/kollect/templates/validating-webhook-configuration.yaml b/charts/kollect/templates/validating-webhook-configuration.yaml index fe78b6f6..209a17e5 100644 --- a/charts/kollect/templates/validating-webhook-configuration.yaml +++ b/charts/kollect/templates/validating-webhook-configuration.yaml @@ -1,4 +1,22 @@ {{- if .Values.webhooks.enabled }} +{{- $hooks := list + (dict "name" "vkollectclusterdatabasesink.kb.io" "kind" "kollectclusterdatabasesink" "resource" "kollectclusterdatabasesinks") + (dict "name" "vkollectclustereventsink.kb.io" "kind" "kollectclustereventsink" "resource" "kollectclustereventsinks") + (dict "name" "vkollectclusterinventory.kb.io" "kind" "kollectclusterinventory" "resource" "kollectclusterinventories") + (dict "name" "vkollectclusterprofile.kb.io" "kind" "kollectclusterprofile" "resource" "kollectclusterprofiles") + (dict "name" "vkollectclusterscope.kb.io" "kind" "kollectclusterscope" "resource" "kollectclusterscopes") + (dict "name" "vkollectclustersnapshotsink.kb.io" "kind" "kollectclustersnapshotsink" "resource" "kollectclustersnapshotsinks") + (dict "name" "vkollectclustertarget.kb.io" "kind" "kollectclustertarget" "resource" "kollectclustertargets") + (dict "name" "vkollectconnectiontest.kb.io" "kind" "kollectconnectiontest" "resource" "kollectconnectiontests") + (dict "name" "vkollectdatabasesink.kb.io" "kind" "kollectdatabasesink" "resource" "kollectdatabasesinks") + (dict "name" "vkollecteventsink.kb.io" "kind" "kollecteventsink" "resource" "kollecteventsinks") + (dict "name" "vkollectinventory.kb.io" "kind" "kollectinventory" "resource" "kollectinventories") + (dict "name" "vkollectprofile.kb.io" "kind" "kollectprofile" "resource" "kollectprofiles") + (dict "name" "vkollectremotecluster.kb.io" "kind" "kollectremotecluster" "resource" "kollectremoteclusters") + (dict "name" "vkollectscope.kb.io" "kind" "kollectscope" "resource" "kollectscopes") + (dict "name" "vkollectsnapshotsink.kb.io" "kind" "kollectsnapshotsink" "resource" "kollectsnapshotsinks") + (dict "name" "vkollecttarget.kb.io" "kind" "kollecttarget" "resource" "kollecttargets") +-}} apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: @@ -10,18 +28,20 @@ metadata: cert-manager.io/inject-ca-from: {{ .Release.Namespace }}/{{ include "kollect.fullname" . }}-serving-cert {{- end }} webhooks: +{{- range $hooks }} - admissionReviewVersions: [v1] clientConfig: service: - name: {{ include "kollect.webhookServiceName" . }} - namespace: {{ .Release.Namespace }} - path: /validate-kollect-dev-v1alpha1-kollectprofile + name: {{ include "kollect.webhookServiceName" $ }} + namespace: {{ $.Release.Namespace }} + path: /validate-kollect-dev-v1alpha1-{{ .kind }} failurePolicy: Fail - name: vkollectprofile.kb.io + name: {{ .name }} rules: - apiGroups: [kollect.dev] apiVersions: [v1alpha1] operations: [CREATE, UPDATE] - resources: [kollectprofiles] + resources: [{{ .resource }}] sideEffects: None {{- end }} +{{- end }} diff --git a/charts/kollect/tests/validating_webhook_configuration_test.yaml b/charts/kollect/tests/validating_webhook_configuration_test.yaml new file mode 100644 index 00000000..4d02bbfe --- /dev/null +++ b/charts/kollect/tests/validating_webhook_configuration_test.yaml @@ -0,0 +1,40 @@ +suite: validating webhook configuration +templates: + - validating-webhook-configuration.yaml +tests: + - it: renders no VWC when webhooks.enabled is false + set: + webhooks: + enabled: false + asserts: + - hasDocuments: + count: 0 + - it: registers all sixteen validating webhooks when enabled + set: + webhooks: + enabled: true + certManager: + create: true + asserts: + - hasDocuments: + count: 1 + - isKind: + of: ValidatingWebhookConfiguration + - equal: + path: metadata.annotations["cert-manager.io/inject-ca-from"] + value: NAMESPACE/RELEASE-NAME-kollect-serving-cert + - lengthEqual: + path: webhooks + count: 16 + - equal: + path: webhooks[0].name + value: vkollectclusterdatabasesink.kb.io + - equal: + path: webhooks[0].clientConfig.service.path + value: /validate-kollect-dev-v1alpha1-kollectclusterdatabasesink + - equal: + path: webhooks[15].name + value: vkollecttarget.kb.io + - equal: + path: webhooks[15].rules[0].resources[0] + value: kollecttargets diff --git a/docs/ASSURANCE-CASE.md b/docs/ASSURANCE-CASE.md index a45f6aa8..38689a3a 100644 --- a/docs/ASSURANCE-CASE.md +++ b/docs/ASSURANCE-CASE.md @@ -21,7 +21,7 @@ Security requirements are tracked as NFR-SEC in [REQUIREMENTS.md](REQUIREMENTS.m | NFR-SEC-2 | Default verify TLS; `insecureSkipVerify` opt-in and surfaced | ADR-0104, sink validators | | NFR-SEC-3 | Tenancy via `KollectScope` + SAR; least-privilege RBAC | ADR-0203, ADR-0704, `task audit:rbac` | | NFR-SEC-4 | Sensitive-key redaction before export | ADR-0303, ADR-0405 | -| NFR-SEC-5 | Distroless nonroot image; minimal attack surface | Dockerfile, ADR-0705 | +| NFR-SEC-5 | Nonroot image (UID 65532); minimal runtime deps (`git`/`openssh-client` for git CLI engine) | Dockerfile, ADR-0705 | ## Trust boundaries diff --git a/docs/OPERATOR-MANUAL.md b/docs/OPERATOR-MANUAL.md index 1294c894..5ee2ce84 100644 --- a/docs/OPERATOR-MANUAL.md +++ b/docs/OPERATOR-MANUAL.md @@ -107,6 +107,19 @@ Walkthrough: [Postgres state store](examples/postgres-state-store.md). TLS trust for sinks uses `caBundle` or `caSecretRef` on the sink spec — same resolution as export and connection probes ([ADR-0403](adr/0403-connection-test.md)). +### Git snapshot sinks (`spec.git.engine`) + +| `spec.git.engine` | Runtime needs | Notes | +| --- | --- | --- | +| `go-git` (default) | None beyond the manager binary | Pure Go transport; works on minimal images | +| `cli` | `git` and `openssh-client` in `PATH` | Native clone/commit/push; SSH uses `GIT_SSH_COMMAND` with `openssh-client` | + +The published operator image (`ghcr.io/konih/kollect`) ships **Debian bookworm-slim** with +`git`, `openssh-client`, and `ca-certificates` on UID/GID **65532**. Default `go-git` export is +unchanged; the image change enables `engine: cli` and full `git ls-remote` connection probes. +Custom images built from an older distroless base must install `git` (and `openssh-client` for SSH) +when using `engine: cli`. + ### Webhook serving certificate Validating webhooks require a TLS serving cert mounted on every manager replica diff --git a/docs/PLATFORM-DECISIONS.md b/docs/PLATFORM-DECISIONS.md index ff2b898c..ae66216e 100644 --- a/docs/PLATFORM-DECISIONS.md +++ b/docs/PLATFORM-DECISIONS.md @@ -52,7 +52,7 @@ backward compatibility. Breaking changes are batched deliberately before a futur 6. Argo **`Application`** sample + **contract test** (contract test **first**; then samples) 7. Hub `mode: hub|spoke` + merge lib (`inprocess`); no hub CRD 8. **`KollectClusterTarget`** — API + webhook only until namespaced MVP proven; controller later -9. **Secondary watches** — Profile → Targets, Sink → Inventories (beta requirement) +9. **Secondary watches** — Profile → Targets, family Sink → Inventories (beta requirement) 10. **Generic CRD sample** — `cert-manager.io/Certificate` + contract test 11. **GitLab sink** — Phase 2 (custom CA via `tls.caSecretRef`; first enterprise presentation path) 12. **Hub ingest** — SAR **`create`** on `kollectremoteclusters` @@ -75,7 +75,7 @@ Sink/transport reframe — [ADR-0401](adr/0401-sink-taxonomy-state-vs-stream.md) | Topic | Decision | | --- | --- | -| Secondary watches | **Ship** — `KollectProfile` change enqueues referring Targets; `KollectSink` change enqueues Inventories with `sinkRefs` | +| Secondary watches | **Ship** — `KollectProfile` change enqueues referring Targets; family sink change enqueues Inventories with matching `snapshotSinkRefs`, `databaseSinkRefs`, or `eventSinkRefs` | | Generic CRD sample | **`cert-manager.io/Certificate`** — contract test first, then profile + target + walkthrough | | `KollectClusterTarget` controller | **Defer** — API + webhook + sample only until namespaced e2e solid | | `profileRef` (cluster target) | Resolves **`KollectProfile` in platform namespace** (Helm `platformNamespace`); `KollectClusterProfile` later | @@ -122,7 +122,7 @@ the corresponding code merges. | Inventory read SAR | **`get`** on `kollectinventories` in caller namespace; **`list`** for index | 1 | | Hub ingest SAR | **`create`** on **`kollectremoteclusters`** in hub namespace | 2 | | GitLab sink | **`type: gitlab`** backend + custom CA TLS; Phase 2 after Git path proven | 2 | -| Secondary watches | Profile → Targets; Sink → Inventories | 1 (beta) | +| Secondary watches | Profile → Targets; family Sink → Inventories | 1 (beta) | | TokenReview/SAR cache | **30s TTL** in-memory per `(token hash, verb, resource)`; flag + `disabled` for dev | 1 | | `maxExportBytes` | Global manager default (~**1.5 MiB**) + optional **`KollectInventory.spec.maxExportBytes`**; webhook rejects override > global cap | 1 | @@ -188,7 +188,9 @@ the corresponding code merges. flowchart TD subgraph teamNs [Team namespace] Prof[KollectProfile] - Sink[KollectSink] + Snap[KollectSnapshotSink] + Db[KollectDatabaseSink] + Ev[KollectEventSink] Scope[KollectScope] Tgt[KollectTarget] Inv[KollectInventory] @@ -199,15 +201,20 @@ flowchart TD Scope -.-> Tgt Scope -.-> Inv Tgt --> Inv - Inv --> Sink - ConnTest -.-> Sink - ConnTest -.-> Prof + Inv --> Snap + Inv --> Db + Inv --> Ev + ConnTest -.-> Snap + ConnTest -.-> Db + ConnTest -.-> Ev ``` | Kind | Scope | Notes | | --- | --- | --- | | `KollectProfile` | Namespace | Same-ns `profileRef` on Target | -| `KollectSink` | **Namespace** | Same-ns `sinkRefs` on Inventory | +| `KollectSnapshotSink` | **Namespace** | Same-ns `snapshotSinkRefs` on Inventory | +| `KollectDatabaseSink` | **Namespace** | Same-ns `databaseSinkRefs` on Inventory | +| `KollectEventSink` | **Namespace** | Same-ns `eventSinkRefs` on Inventory | | `KollectTarget` | Namespace | Default for `tenantMode`; same-ns `profileRef` | | `KollectClusterTarget` | **Cluster** | Platform operator; `namespaceSelector` + `KollectClusterProfile` ref | | `KollectInventory` | Namespace | Aggregates namespaced targets in namespace | diff --git a/docs/QUICKSTART.md b/docs/QUICKSTART.md index 9741468f..d6abe860 100644 --- a/docs/QUICKSTART.md +++ b/docs/QUICKSTART.md @@ -103,9 +103,10 @@ live API access. | File | Kind | Role | | --- | --- | --- | | `kollect_v1alpha1_kollectprofile.yaml` | `KollectProfile` | Extract Deployment image + labels | -| `kollect_v1alpha1_kollectsink.yaml` | `KollectSink` | Git sink (example repo URL) | +| `kollect_v1alpha1_kollectdatabasesink.yaml` | `KollectDatabaseSink` | Postgres sink (portal SoR) | +| `kollect_v1alpha1_kollectsnapshotsink.yaml` | `KollectSnapshotSink` | Git snapshot sink (audit) | | `kollect_v1alpha1_kollecttarget.yaml` | `KollectTarget` | Collect Deployments using the profile | -| `kollect_v1alpha1_kollectinventory.yaml` | `KollectInventory` | Aggregate and export to the sink | +| `kollect_v1alpha1_kollectinventory.yaml` | `KollectInventory` | Aggregate and export to family sinks | Narrative walkthrough with expected behavior notes: [examples/deployment-inventory.md](examples/deployment-inventory.md). @@ -135,28 +136,37 @@ Set `watchMode: OptIn` to collect only explicitly `enabled` namespaces/resources Sample opt-in target: `config/samples/kollect_v1alpha1_kollecttarget_opt-in.yaml`. -### Connection test (sink) +### Connection test (family sinks) -Samples set `spec.connectionTest: true` on `KollectSink`. The operator probes Git/Postgres/Kafka -and sets **`ConnectionVerified`** ([ADR-0403](adr/0403-connection-test.md)). +Samples set `spec.connectionTest: true` on **`KollectDatabaseSink`** and **`KollectSnapshotSink`**. +The operator probes Postgres/Git (and other wired backends) and sets **`ConnectionVerified`** +([ADR-0403](adr/0403-connection-test.md), [ADR-0414](adr/0414-sink-family-crds.md)). ```sh -kubectl wait --for=condition=ConnectionVerified kollectsink/git-inventory \ +kubectl wait --for=condition=ConnectionVerified kollectdatabasesink/postgres-inventory-demo \ + -n default --timeout=60s +kubectl describe kollectdatabasesink postgres-inventory-demo -n default +``` + +Git snapshot sink: + +```sh +kubectl wait --for=condition=ConnectionVerified kollectsnapshotsink/git-inventory-demo \ -n default --timeout=60s -kubectl describe kollectsink git-inventory -n default ``` Re-test without editing spec: ```sh -kubectl annotate kollectsink git-inventory -n default kollect.dev/test-connection=true --overwrite +kubectl annotate kollectsnapshotsink git-inventory-demo -n default \ + kollect.dev/test-connection=true --overwrite ``` ### Optional: Git credentials -The sample Git sink references a placeholder repository. For real exports, create a Secret with -credentials and point `spec.secretRef` at it. Connection tests can pass without a writable remote; -export requires valid credentials and endpoint reachability. +The sample Git snapshot sink references a placeholder repository. For real exports, create a Secret +with credentials and point `spec.secretRef` at it. Connection tests can pass without a writable +remote; export requires valid credentials and endpoint reachability. ## Verify @@ -175,7 +185,7 @@ Postgres row counts, etc. — not full payloads; see [ADR-0103](adr/0103-etcd-li ### CR status ```sh -kubectl get kollectprofiles,kollectsinks,kollectinventories +kubectl get kprof,ksnap,kdb,kinv -A kubectl get kollecttargets -A kubectl describe kollectinventory -n default team-inventory ``` diff --git a/docs/adr/0101-kubebuilder-v4.md b/docs/adr/0101-kubebuilder-v4.md index adcf3a48..185169dd 100644 --- a/docs/adr/0101-kubebuilder-v4.md +++ b/docs/adr/0101-kubebuilder-v4.md @@ -23,7 +23,7 @@ patterns — thin reconcilers, workqueues, conditions, and leader election. - **Layout:** standard Kubebuilder v4 project structure (`api/v1alpha1`, `internal/{controller,collect,sink}`, `cmd/main.go`, `config/`, `charts/kollect/`). - **API version:** `kollect.dev/v1alpha1` (alpha until export/doc flows stabilize). -- **Image:** distroless nonroot; evaluate `ko` for reproducible builds (see PLAN Phase 0 tooling). +- **Image:** Debian bookworm-slim nonroot (UID 65532) with `git`/`openssh-client` for git CLI engine; evaluate `ko` for reproducible builds (see PLAN Phase 0 tooling). ## Consequences diff --git a/docs/adr/0403-connection-test.md b/docs/adr/0403-connection-test.md index c7675d12..c7bad272 100644 --- a/docs/adr/0403-connection-test.md +++ b/docs/adr/0403-connection-test.md @@ -12,11 +12,12 @@ mechanisms remain valid. Product requirements call for a **first-class connection test** with clear, discoverable feedback when sinks are misconfigured ([REQUIREMENTS.md](../REQUIREMENTS.md), [engineering guidelines](../development/guidelines.md)). -[ADR-0202](0202-static-vs-reconciled.md) originally assumed static `KollectSink` objects with **no +[ADR-0202](0202-static-vs-reconciled.md) originally assumed static sink config objects with **no controller**, probing via annotations and surfacing `SinkReachable` on reconciled -`KollectTarget` / `KollectInventory`. Implementation added a **minimal `KollectSink` reconciler** -that runs connectivity checks and writes `ConnectionVerified` on the sink -(`internal/controller/kollectsink_controller.go`). +`KollectTarget` / `KollectInventory`. Implementation added **minimal family sink reconcilers** +(`KollectSnapshotSink`, `KollectDatabaseSink`, `KollectEventSink`) that run connectivity checks +and write `ConnectionVerified` on the sink +(`internal/controller/family_sink_connection.go`). An alternative is a dedicated **`KollectConnectionTest`** CR (apply a test object, wait on status, garbage-collect). That pattern helps composite or cross-cluster probes but adds API surface, RBAC, @@ -27,7 +28,7 @@ webhooks, and orphan CR lifecycle. ### ~~Reject `KollectConnectionTest` CR~~ → **Accepted in ADR-0703** Add namespaced **`KollectConnectionTest`** for audited/CI/composite probes. Keep **declarative -spec** + **imperative annotation** on `KollectSink` for quick sink-only retests, plus pipeline +spec** + **imperative annotation** on family sink CRDs for quick sink-only retests, plus pipeline conditions on reconciled objects. ### Sink connectivity (implemented) @@ -39,16 +40,20 @@ conditions on reconciled objects. | **Annotation `kollect.dev/test-connection: "true"`** | One-shot re-test without editing spec | Probe uses the same TLS trust and secret resolution as export (`caBundle` / `caSecretRef`, -`secretRef`). Supported types today: +`secretRef`). Supported types today (per family CRD): -| `spec.type` | Probe wired | -| --- | --- | -| `git`, `gitlab`, `postgres`, `kafka`, `s3` | ✅ | -| `nats`, `gcs` | ⬜ not wired ([kollectsink.md](../crds/kollectsink.md)) | +| Family CRD | `spec.type` | Probe wired | +| --- | --- | --- | +| `KollectSnapshotSink` | `git`, `gitlab`, `s3`, `gcs` | ✅ | +| `KollectDatabaseSink` | `postgres` | ✅ | +| `KollectEventSink` | `kafka`, `nats` | ✅ | + +Stub backends (`azureblob`, `http`, `bigquery`) register in the sink registry but return *not +implemented* at probe/export time until shipped. Extend per sink as backends mature. -**Status on `KollectSink`:** +**Status on family sink CRDs:** | Condition | Meaning | | --- | --- | @@ -62,14 +67,14 @@ Extend per sink as backends mature. **`kubectl` example:** ```sh -kubectl wait --for=condition=ConnectionVerified kollectsink/git-inventory \ +kubectl wait --for=condition=ConnectionVerified kollectsnapshotsink/git-inventory-demo \ -n default --timeout=60s ``` Re-run without spec change: ```sh -kubectl annotate kollectsink git-inventory -n kollect-system \ +kubectl annotate kollectsnapshotsink git-inventory-demo -n default \ kollect.dev/test-connection=true --overwrite ``` @@ -82,8 +87,8 @@ End-to-end export health belongs on **reconciled** objects, not only the sink: | **`SinkReachable`** | `KollectInventory`, `KollectTarget` | Sink resolution (`ConnectionVerified` / sink found) before export; **`ExportSucceeded`** / **`ExportFailed`** after inventory export attempts. `Synced` remains the primary export condition per [ADR-0602](0602-error-taxonomy.md). | `KollectTarget` derives sink refs from **`KollectInventory` in the same namespace** (targets have no -direct `sinkRefs`). Inventory reconciler watches **`KollectSink`** status changes to requeue affected -inventories. +direct family sink refs). Inventory reconciler watches **family sink** status changes to requeue +affected inventories. Sink-only `ConnectionVerified` proves **credentials and network to the backend**; it does not prove the full collect → aggregate → export path. @@ -92,8 +97,9 @@ prove the full collect → aggregate → export path. `KollectProfile` and `KollectScope` remain **webhook-validated static config** (no controller). -`KollectSink` has a **narrow reconciler** whose sole job is connection test status — not -collection or export. This is an intentional exception to full static-config purity, documented in +Family sink CRDs (`KollectSnapshotSink`, `KollectDatabaseSink`, `KollectEventSink`) each have a +**narrow reconciler** whose sole job is connection test status — not collection or export. This is +an intentional exception to full static-config purity, documented in [ADR-0202](0202-static-vs-reconciled.md). ### `KollectConnectionTest` CR (ADR-0703) diff --git a/docs/adr/0705-release-supply-chain.md b/docs/adr/0705-release-supply-chain.md index 0876a841..00b25d13 100644 --- a/docs/adr/0705-release-supply-chain.md +++ b/docs/adr/0705-release-supply-chain.md @@ -73,7 +73,7 @@ are **documented and deferred** — not enabled — so one maintainer can ship w | --- | ---: | --- | --- | | Dangerous-Workflow | 0 critical | **Done** | No `pull_request_target`; workflow inputs passed via env vars; actions SHA-pinned | | Token-Permissions | 0 high | **Done** | Default `contents: read`; `security-events: write` scoped to CodeQL/Scorecard analyze jobs only; release job documents why `contents: write` is required | -| Pinned-Dependencies | 0 medium | **Done** | Actions pinned to commit SHA; distroless base image by digest; Helm tarball SHA256-verified; pip docs deps hash-locked (`--require-hashes`) | +| Pinned-Dependencies | 0 medium | **Done** | Actions pinned to commit SHA; runtime base image by digest (Debian bookworm-slim); Helm tarball SHA256-verified; pip docs deps hash-locked (`--require-hashes`) | | SAST | 0 medium | **Done** | `golangci-lint` + `govulncheck` in CI; **CodeQL** for Go (`.github/workflows/codeql.yaml`) | | Vulnerabilities | 0 | **Done** | `govulncheck` on every PR; grpc ≥1.79.3 and otel/sdk ≥1.43.0; Trivy gates release images; Dependabot alerts enabled | | Security-Policy | 10 | **Done** | `SECURITY.md` | diff --git a/docs/crds/kollectsnapshotsink.md b/docs/crds/kollectsnapshotsink.md index 0a60a3d3..86a642b4 100644 --- a/docs/crds/kollectsnapshotsink.md +++ b/docs/crds/kollectsnapshotsink.md @@ -17,6 +17,7 @@ snapshot sinks via `KollectInventory.spec.snapshotSinkRefs`. | `spec.type` | Backend: `git`, `gitlab`, `s3`, `gcs`, `azureblob`, `http` | | `spec.endpoint` | Repository URL, bucket URI, or webhook URL | | `spec.git` / `spec.gitlab` / `spec.objectStore` / `spec.http` | Type-specific settings | +| `spec.git.engine` | Git export backend: `go-git` (default, pure Go) or `cli` (native `git` binary). `cli` is required for some SSH/KEX edge cases; shipped operator image includes `git` and `openssh-client` | | `spec.exportMinInterval` | Default per-ref debounce when inventory ref omits override | | `spec.connectionTest` | Automatic probe on create/update (default `true`) | diff --git a/docs/crds/kollecttarget.md b/docs/crds/kollecttarget.md index 3915ab57..928fe82d 100644 --- a/docs/crds/kollecttarget.md +++ b/docs/crds/kollecttarget.md @@ -101,7 +101,7 @@ kubectl apply -f config/samples/kollect_v1alpha1_kollecttarget_helm-releases.yam | `Ready=True` | Collecting | Profile resolved; informer registered | None | | `Synced=True` | Healthy | `reason`: `Collecting` with item count | Monitor `status` message | | `Degraded=True` | Blocked | See `reason` below | Fix root cause; generation bump re-reconciles | -| `SinkReachable=True/False` | Export path | Namespace inventory sinks reachable | Fix [KollectSink](kollectsink.md) connection | +| `SinkReachable=True/False` | Export path | Namespace inventory family sinks reachable | Fix [KollectSnapshotSink](kollectsnapshotsink.md), [KollectDatabaseSink](kollectdatabasesink.md), or [KollectEventSink](kollecteventsink.md) connection | ### Common `Degraded` reasons @@ -122,7 +122,7 @@ kubectl apply -f config/samples/kollect_v1alpha1_kollecttarget_helm-releases.yam | Actor | Verbs | Resource | Notes | | --- | --- | --- | --- | | Team engineers | `get`, `list`, `watch`, `create`, `update`, `patch`, `delete` | `kollecttargets` | Tenant namespace | -| Operator | `get`, `list`, `watch` | `kollecttargets`, `kollectprofiles`, `kollectscopes`, `kollectsinks` | Reconcile | +| Operator | `get`, `list`, `watch` | `kollecttargets`, `kollectprofiles`, `kollectscopes`, `kollectsnapshotsinks`, `kollectdatabasesinks`, `kollecteventsinks` | Reconcile | | Operator | `get`, `list`, `watch` | Target GVK resources | Dynamic — per profile (e.g. `deployments`) | | Operator | `update`, `patch` | `kollecttargets/status` | Write conditions | diff --git a/docs/development/coding-standards.md b/docs/development/coding-standards.md index 4acea607..56d7af45 100644 --- a/docs/development/coding-standards.md +++ b/docs/development/coding-standards.md @@ -74,8 +74,10 @@ Short, actionable rules for Go code in this repo. Operator reconcile semantics a ### Container builds -- **MUST** build the operator manager with `CGO_ENABLED=0` for distroless images - ([`Dockerfile`](../../Dockerfile)); enable CGO locally only for `task coverage:race`. +- **MUST** build the operator manager with `CGO_ENABLED=0` for the shipped image + ([`Dockerfile`](../../Dockerfile)); enable CGO locally only for `task coverage:race`. The runtime + stage is Debian bookworm-slim (nonroot UID 65532) with `git` and `openssh-client` for + `spec.git.engine: cli` and `git ls-remote` probes; `go-git` (default) does not use those binaries. ## Go style and lint diff --git a/docs/development/guidelines.md b/docs/development/guidelines.md index 255f5d1d..1dc1163d 100644 --- a/docs/development/guidelines.md +++ b/docs/development/guidelines.md @@ -49,7 +49,7 @@ Operator-specific error taxonomy drives reconcile behavior. For Go wrapping conv - **Least-privilege RBAC** — minimal generated roles; SAR pre-check before list/watch. - **Tenancy** — optional `KollectScope` (future) for allowed GVKs, namespaces, sinks. - **Secrets** — credentials only via `secretRef`; never in spec/status or logs. -- **Container hardening** — distroless non-root, read-only rootfs, dropped capabilities, seccomp. +- **Container hardening** — non-root runtime image (UID 65532), read-only rootfs, dropped capabilities, seccomp. - **Network** — restrictive `NetworkPolicy` for production egress. - **Transport** — TLS verification required for sink and doc endpoints; support org **custom CA** (no disable-verify in prod). - **Input validation** — CEL in CRD OpenAPI + **validating webhooks before reconcile workarounds**. diff --git a/docs/examples/connection-test.md b/docs/examples/connection-test.md index 1c3ce63d..1f5094f0 100644 --- a/docs/examples/connection-test.md +++ b/docs/examples/connection-test.md @@ -1,67 +1,128 @@ # Example: Connection test !!! tip "Opt out in production" - `spec.connectionTest` defaults to **true**. Set `connectionTest: false` when automatic probes on - every spec change are undesirable. Use the `kollect.dev/test-connection` annotation or a - `KollectConnectionTest` CR for on-demand probes. + `spec.connectionTest` defaults to **true** on family sink CRDs. Set `connectionTest: false` when + automatic probes on every spec change are undesirable. Use the `kollect.dev/test-connection` + annotation or a `KollectConnectionTest` CR for on-demand probes. Kollect offers **three** ways to verify sink connectivity before relying on export -([ADR-0403](../adr/0403-connection-test.md), [ADR-0703](../adr/0703-platform-architecture-pivot.md)). +([ADR-0403](../adr/0403-connection-test.md), [ADR-0414](../adr/0414-sink-family-crds.md), +[ADR-0703](../adr/0703-platform-architecture-pivot.md)). ## Overview | Mechanism | Best for | Writes to | | --- | --- | --- | -| `spec.connectionTest` (default **true**) | Automatic probe on create/update | `KollectSink.status` | -| `kollect.dev/test-connection` annotation | On-demand prod re-test | `KollectSink.status` | +| `spec.connectionTest` (default **true**) | Automatic probe on create/update | Family sink `status` | +| `kollect.dev/test-connection` annotation | On-demand prod re-test | Family sink `status` | | `KollectConnectionTest` CR | Audited / CI pipelines | `KollectConnectionTest.status` | **Default:** `spec.connectionTest: true` (CRD OpenAPI default). Set `false` to opt out. +Family sink CRDs ([ADR-0414](../adr/0414-sink-family-crds.md)): + +| CRD | Role | Wired probe types | +| --- | --- | --- | +| `KollectSnapshotSink` | Snapshot store | `git`, `gitlab`, `s3`, `gcs` | +| `KollectDatabaseSink` | Relational SoR | `postgres` | +| `KollectEventSink` | Event emitter | `kafka`, `nats` | + +Stub backends (`azureblob`, `http`, `bigquery`) pass admission but return *not implemented* at probe +time until shipped. + ## Sink probe — `spec.connectionTest` -`config/samples/kollect_v1alpha1_kollectsink_postgres.yaml` sets `connectionTest: true`. +`config/samples/kollect_v1alpha1_kollectdatabasesink.yaml` sets `connectionTest: true`. + +```sh +kubectl apply -f config/samples/kollect_v1alpha1_kollectdatabasesink.yaml +kubectl wait --for=condition=ConnectionVerified kollectdatabasesink/postgres-inventory-demo \ + -n default --timeout=60s +kubectl describe kdb postgres-inventory-demo -n default +``` + +Git snapshot sink sample (`config/samples/kollect_v1alpha1_kollectsnapshotsink.yaml`): ```sh -kubectl wait --for=condition=ConnectionVerified kollectsink/postgres-inventory-demo \ +kubectl wait --for=condition=ConnectionVerified kollectsnapshotsink/git-inventory-demo \ -n default --timeout=60s ``` -Supported probe types: `git`, `gitlab`, `postgres`, `kafka`, `s3`. +Each family has a **narrow reconciler** whose sole job is connection-test status — not collection or +export ([ADR-0403](../adr/0403-connection-test.md)). ## Annotation re-test +Trigger a one-shot probe without editing `spec`: + +```sh +kubectl annotate kollectdatabasesink postgres-inventory-demo -n default \ + kollect.dev/test-connection=true --overwrite +``` + ```sh -kubectl annotate kollectsink postgres-inventory-demo -n default \ +kubectl annotate kollectsnapshotsink git-inventory-demo -n default \ kollect.dev/test-connection=true --overwrite ``` +When `spec.connectionTest: false`, the annotation is the supported way to re-probe in production. +After a successful annotation-only probe, the reconciler clears the annotation (kept when +`spec.connectionTest: true`). + ## KollectConnectionTest CR -`config/samples/kollect_v1alpha1_kollectconnectiontest.yaml`: +`config/samples/kollect_v1alpha1_kollectconnectiontest.yaml` — `spec.sinkRef` names exactly one +family sink: ```yaml spec: - sinkRef: postgres-inventory-demo + sinkRef: + databaseSinkRef: postgres-inventory-demo ownerSink: true ``` +Snapshot or event probes use `snapshotSinkRef` or `eventSinkRef` instead. + ```sh kubectl apply -f config/samples/kollect_v1alpha1_kollectconnectiontest.yaml kubectl wait --for=condition=ConnectionVerified kollectconnectiontest/postgres-sink-probe \ -n default --timeout=120s +kubectl get kconntest postgres-sink-probe -n default -o wide ``` -Default `spec.ttlSecondsAfterFinished`: **300** (CR auto-deleted after probe). +Default `spec.ttlSecondsAfterFinished`: **300** (CR auto-deleted after probe completes + TTL). + +Re-run after fixing credentials: delete and re-apply the CR, or patch `spec` to bump generation. + +## Status conditions + +On family sink CRDs: + +| Condition | Meaning | +| --- | --- | +| **`ConnectionVerified`** `True` | Last probe succeeded | +| **`ConnectionVerified`** `False` | Probe failed (`ConnectionTestFailed`, `SecretResolveFailed`, …) | +| **`Degraded`** `True` | Set alongside failed probe | +| **`TLSInsecure`** `True` | `insecureSkipVerify` enabled (dev warning) | + +Operator metric: `kollect_sink_connection_test_total{type,result}`. ## Pipeline reachability -`ConnectionVerified` proves backend credentials/network. `SinkReachable` and `Synced` on -`KollectInventory` prove export path health — see -[Deployment inventory](deployment-inventory.md). +`ConnectionVerified` on a family sink proves **credentials and network to the backend**. End-to-end +export health belongs on reconciled objects: + +| Condition | Object | Meaning | +| --- | --- | --- | +| **`SinkReachable`** | `KollectInventory`, `KollectTarget` | Family sink resolution before export | +| **`Synced`** | `KollectInventory` | Export succeeded for a sink ref | + +See [Deployment inventory](deployment-inventory.md) for the full Profile → Target → Inventory → +family sink path. ## Related - [KollectConnectionTest](../crds/kollectconnectiontest.md) +- [KollectSnapshotSink](../crds/kollectsnapshotsink.md) · [KollectDatabaseSink](../crds/kollectdatabasesink.md) · [KollectEventSink](../crds/kollecteventsink.md) - [Postgres state store](postgres-state-store.md) -- [ADR-0403](../adr/0403-connection-test.md) +- [ADR-0403](../adr/0403-connection-test.md) · [ADR-0414](../adr/0414-sink-family-crds.md) diff --git a/docs/examples/deployment-inventory.md b/docs/examples/deployment-inventory.md index 794676f7..36d40f8b 100644 --- a/docs/examples/deployment-inventory.md +++ b/docs/examples/deployment-inventory.md @@ -5,10 +5,11 @@ [Kind local lab](kind-local-lab.md)) and can apply manifests from `config/samples/`. The default path uses a Postgres sink; swap to Git for audit-only workflows. -This walkthrough connects the four core **namespaced** CRDs into a minimal pipeline: define **what** -to extract (`KollectProfile`), **where** to send it (`KollectSink`), **which** resources to watch -(`KollectTarget`), and **when** to aggregate and export (`KollectInventory`). There is **no -`KollectHub` CRD** — hub aggregation uses Helm `mode: hub` ([ADR-0703](../adr/0703-platform-architecture-pivot.md)). +This walkthrough connects the core **namespaced** CRDs into a minimal pipeline: define **what** +to extract (`KollectProfile`), **where** to send it (family sinks — `KollectDatabaseSink`, +`KollectSnapshotSink`), **which** resources to watch (`KollectTarget`), and **when** to aggregate +and export (`KollectInventory`). There is **no `KollectHub` CRD** — hub aggregation uses Helm +`mode: hub` ([ADR-0703](../adr/0703-platform-architecture-pivot.md)). **Default sample path:** Postgres state store (`postgres-inventory-demo`). Swap to `git-inventory-demo` for Git audit/CI. See [Postgres state store](postgres-state-store.md) and @@ -23,14 +24,17 @@ flowchart LR Profile[KollectProfile
Deployment schema] Target[KollectTarget
select Deployments] Inv[KollectInventory
aggregate + export] - Sink[KollectSink
Postgres or Git] + Db[KollectDatabaseSink
Postgres] + Snap[KollectSnapshotSink
Git] K8s[(Kubernetes API)] Profile --> Target Target --> K8s Target --> Inv - Inv --> Sink - Sink --> Store[(Postgres / Git)] + Inv --> Db + Inv --> Snap + Db --> PG[(Postgres)] + Snap --> Git[(Git repo)] ``` ## Scale @@ -95,20 +99,21 @@ Set `type: list` on multi-value attributes. CEL equivalent: See [DATA-FLOWS.md](../DATA-FLOWS.md#3-attribute-extraction-jsonpath-arrays) and [ADR-0302](../adr/0302-cel-jsonpath-extraction.md). -## Step 2 — KollectSink +## Step 2 — Family sinks !!! warning "Same-namespace sink refs" - `KollectSink` is **namespaced** — create sinks in the same namespace as `KollectInventory` - `sinkRefs`. Cross-namespace references fail admission with `SinkNotFound` - ([ADR-0703](../adr/0703-platform-architecture-pivot.md)). + Family sink CRDs are **namespaced** — create sinks in the same namespace as + `KollectInventory` family ref lists (`snapshotSinkRefs`, `databaseSinkRefs`, `eventSinkRefs`). + Cross-namespace references fail admission with `SinkNotFound` + ([ADR-0414](../adr/0414-sink-family-crds.md)). -### Postgres (default sample) +### KollectDatabaseSink (default sample — Postgres) -`config/samples/kollect_v1alpha1_kollectsink_postgres.yaml` +`config/samples/kollect_v1alpha1_kollectdatabasesink.yaml` ```yaml apiVersion: kollect.dev/v1alpha1 -kind: KollectSink +kind: KollectDatabaseSink metadata: name: postgres-inventory-demo namespace: default @@ -126,13 +131,13 @@ spec: Create the DSN secret before export — see [Postgres state store](postgres-state-store.md). -### Git (audit / CI) +### KollectSnapshotSink (Git audit / CI) -`config/samples/kollect_v1alpha1_kollectsink.yaml` +`config/samples/kollect_v1alpha1_kollectsnapshotsink.yaml` ```yaml apiVersion: kollect.dev/v1alpha1 -kind: KollectSink +kind: KollectSnapshotSink metadata: name: git-inventory-demo namespace: default @@ -140,16 +145,19 @@ spec: type: git endpoint: https://github.com/konih/kollect-inventory-demo.git connectionTest: true + git: + branch: main + pushPolicy: Commit + commitMessage: "chore(inventory): export {namespace}/{name}" # secretRef: # name: git-push-credentials # namespace: kollect-system ``` -**Behavior:** the inventory controller resolves sinks via the registry (`git`, `postgres`, `kafka`, -`gitlab`, `s3`, `gcs`). With `connectionTest: true`, the operator probes on create/update and sets -`ConnectionVerified` on the sink. Export commits deterministic JSON snapshots to Git or upserts rows -to Postgres; status stores summary refs (commit SHA), not the full payload -([ADR-0103](../adr/0103-etcd-limit.md)). +**Behavior:** the inventory controller resolves family sinks via the registry. With +`connectionTest: true`, family sink reconcilers probe on create/update and set `ConnectionVerified`. +Export commits deterministic JSON snapshots to Git or upserts rows to Postgres; status stores +summary refs (commit SHA), not the full payload ([ADR-0103](../adr/0103-etcd-limit.md)). Production installs should set `connectionTest: false` (chart default) and re-probe on demand — see [Connection test](connection-test.md). @@ -207,7 +215,7 @@ kubectl label deployment nginx app.kubernetes.io/name=nginx --overwrite ## Step 4 — KollectInventory -Namespaced aggregator (same namespace as targets) referencing one or more sinks. +Namespaced aggregator (same namespace as targets) referencing family sinks. `config/samples/kollect_v1alpha1_kollectinventory.yaml` @@ -219,8 +227,9 @@ metadata: namespace: default spec: exportMinInterval: 30s - sinkRefs: + databaseSinkRefs: - postgres-inventory-demo + snapshotSinkRefs: - name: git-inventory-demo exportMinInterval: 1h suspend: false @@ -232,7 +241,7 @@ spec: ref → sink default → inventory default → scope floor — see [ADR-0413](../adr/0413-export-interval-scheduling.md). -For Git-only audit, use a single string ref: `sinkRefs: [git-inventory-demo]`. +For Git-only audit, use a single snapshot ref: `snapshotSinkRefs: [git-inventory-demo]`. **Behavior:** @@ -246,13 +255,13 @@ For Git-only audit, use a single string ref: `sinkRefs: [git-inventory-demo]`. ```sh kubectl apply -k config/samples/ -kubectl get kprof,ksink,ktgt,kinv -A +kubectl get kinv,ktgt,ksnap,kdb -A ``` Verify sink connectivity before relying on export: ```sh -kubectl wait --for=condition=ConnectionVerified kollectsink/postgres-inventory-demo \ +kubectl wait --for=condition=ConnectionVerified kollectdatabasesink/postgres-inventory-demo \ -n default --timeout=60s kubectl describe kollectinventory team-inventory -n default ``` @@ -268,8 +277,8 @@ kubectl describe kollectinventory team-inventory -n default | --- | --- | | Target not found | `KollectTarget` is namespaced — ensure namespace matches | | Profile not found | `profileRef` must name a `KollectProfile` in the **same namespace** as the Target | -| Sink not found | `sinkRefs` must name a `KollectSink` in the **same namespace** as the Inventory | -| No export | Missing DSN/`secretRef`, `ConnectionVerified=False`, or `SinkReachable=False` with reason `SinkNotFound` / `SinkUnreachable` — see `kubectl describe kollectsink` and inventory `status.conditions` | +| Sink not found | Family ref must name a sink in the **same namespace** as the Inventory (`snapshotSinkRefs`, `databaseSinkRefs`, or `eventSinkRefs`) | +| No export | Missing DSN/`secretRef`, `ConnectionVerified=False`, or `SinkReachable=False` with reason `SinkNotFound` / `SinkUnreachable` — see `kubectl describe kollectdatabasesink` / `kollectsnapshotsink` and inventory `status.conditions` | | Empty item count | No Deployments match selector, or target suspended / scope denied | | Namespace skipped | `kollect.dev/namespace-watch: disabled` or `watchMode: OptIn` without `enabled` label | @@ -281,7 +290,8 @@ See [Kind local lab](kind-local-lab.md), [QUICKSTART.md](../QUICKSTART.md), and - [Spoke cluster inventory](spoke-cluster-inventory.md) — Helm `mode: single` install narrative - [Postgres state store](postgres-state-store.md) — DSN secret and delete reconciliation - [Connection test](connection-test.md) — `ConnectionVerified` and `KollectConnectionTest` -- [KollectProfile](../crds/kollectprofile.md) · [KollectSink](../crds/kollectsink.md) · +- [KollectProfile](../crds/kollectprofile.md) · [KollectSnapshotSink](../crds/kollectsnapshotsink.md) · + [KollectDatabaseSink](../crds/kollectdatabasesink.md) · [KollectEventSink](../crds/kollecteventsink.md) · [KollectTarget](../crds/kollecttarget.md) · [KollectInventory](../crds/kollectinventory.md) - [CR reference](../CR-REFERENCE.md) - [ADR-0703: Platform architecture pivot](../adr/0703-platform-architecture-pivot.md) diff --git a/docs/examples/kind-local-lab.md b/docs/examples/kind-local-lab.md index 20b3ef2c..03824860 100644 --- a/docs/examples/kind-local-lab.md +++ b/docs/examples/kind-local-lab.md @@ -38,7 +38,7 @@ Production values and upgrades: [Operator manual](../OPERATOR-MANUAL.md). !!! note "No Postgres required" - For a minimal e2e without a database, apply `config/samples/e2e/team-inventory.yaml` (`sinkRefs: []`). + For a minimal e2e without a database, apply `config/samples/e2e/team-inventory.yaml` (no family sink refs). See [QUICKSTART.md](../QUICKSTART.md) and the [Examples index](README.md). !!! tip "Full showcase demo" diff --git a/docs/examples/kollect-inventory-demo.md b/docs/examples/kollect-inventory-demo.md index 77e4345a..83f62c97 100644 --- a/docs/examples/kollect-inventory-demo.md +++ b/docs/examples/kollect-inventory-demo.md @@ -46,11 +46,11 @@ kubectl apply -f config/samples/kollect_v1alpha1_kollectprofile.yaml kubectl apply -f config/samples/kollect_v1alpha1_kollecttarget.yaml ``` -Git sink + inventory (same namespace as `sinkRefs`): +Git snapshot sink + inventory (same namespace as family sink refs): ```yaml apiVersion: kollect.dev/v1alpha1 -kind: KollectSink +kind: KollectSnapshotSink metadata: name: git-inventory-demo namespace: default @@ -68,7 +68,7 @@ metadata: name: team-inventory namespace: default spec: - sinkRefs: + snapshotSinkRefs: - git-inventory-demo suspend: false ``` diff --git a/docs/examples/spoke-cluster-inventory.md b/docs/examples/spoke-cluster-inventory.md index 3a33451a..d596f4a3 100644 --- a/docs/examples/spoke-cluster-inventory.md +++ b/docs/examples/spoke-cluster-inventory.md @@ -52,7 +52,7 @@ Set `spec.cluster` on the sink for shared-database fan-in across spokes. Full walkthrough: [Deployment inventory](deployment-inventory.md). -E2e without live backends: `config/samples/e2e/team-inventory.yaml` (`sinkRefs: []`). +E2e without live backends: `config/samples/e2e/team-inventory.yaml` (no family sink refs). ## Hub-and-spoke upgrade diff --git a/docs/index.md b/docs/index.md index daf81321..625e4565 100644 --- a/docs/index.md +++ b/docs/index.md @@ -70,8 +70,9 @@ documented in [ADR-0401](adr/0401-sink-taxonomy-state-vs-stream.md), not repeate ## The resource model A pipeline is just a handful of Kubernetes resources: **config you declare** (`KollectProfile`, -`KollectSink`, `KollectScope`) and **objects the operator reconciles** (`KollectTarget`, -`KollectInventory`). Cluster-scoped `KollectCluster*` variants add cross-namespace rollup. +family sinks — `KollectSnapshotSink`, `KollectDatabaseSink`, `KollectEventSink`, `KollectScope`) +and **objects the operator reconciles** (`KollectTarget`, `KollectInventory`). Cluster-scoped +`KollectCluster*` variants add cross-namespace rollup. ```mermaid flowchart LR @@ -81,7 +82,9 @@ flowchart LR direction TB Profile["KollectProfile
what to extract"] Scope["KollectScope
guardrails"] - Sink["KollectSink
where to send"] + Snap["KollectSnapshotSink
snapshot store"] + Db["KollectDatabaseSink
relational SoR"] + Ev["KollectEventSink
event emitter"] end subgraph run["Operator reconciles"] @@ -92,9 +95,9 @@ flowchart LR subgraph out["Sink projections — choose any"] direction TB - Snap["Git · GitLab · S3 · GCS
snapshot store"] + SnapOut["Git · GitLab · S3 · GCS
snapshot store"] Rel["Postgres
relational SoR"] - Evt["NATS · Kafka
event emitter"] + EvtOut["NATS · Kafka
event emitter"] end K8s -- "informer per GVK" --> Target @@ -102,27 +105,31 @@ flowchart LR Target --> Inv Scope -. gates .-> Target Scope -. gates .-> Inv - Inv --> Sink - Sink --> Snap - Sink --> Rel - Sink --> Evt + Inv --> Snap + Inv --> Db + Inv --> Ev + Snap --> SnapOut + Db --> Rel + Ev --> EvtOut classDef api fill:#1F2937,stroke:#6B7280,color:#fff; classDef config fill:#326CE5,stroke:#1b3a8c,color:#fff; classDef work fill:#18B6A3,stroke:#0e6f63,color:#fff; classDef proj fill:#7FB3FF,stroke:#326CE5,color:#081A4B; - class Profile,Scope,Sink config; + class Profile,Scope,Snap,Db,Ev config; class Target,Inv work; - class Snap,Rel,Evt proj; + class SnapOut,Rel,EvtOut proj; ``` | Kind | You set | Role | | --- | --- | --- | | `KollectProfile` | GVK + CEL / JSONPath attributes | **What to extract** from each object | | `KollectTarget` | selectors + `profileRef` | **What to watch** and collect | -| `KollectInventory` | `sinkRefs` + cadence | **Aggregate, debounce, and export** | -| `KollectSink` | type + endpoint + `secretRef` | **Where to send** (Git, Postgres, Kafka, …) | +| `KollectInventory` | family sink refs + cadence | **Aggregate, debounce, and export** | +| `KollectSnapshotSink` | type + endpoint + `secretRef` | **Snapshot store** (Git, S3, GCS, …) | +| `KollectDatabaseSink` | type + credentials | **Relational SoR** (Postgres, …) | +| `KollectEventSink` | type + brokers | **Event emitter** (NATS, Kafka) | | `KollectScope` | allowed GVKs / namespaces / sinks | **Guardrails** for the team namespace | Full fields: [CR reference](CR-REFERENCE.md) · model rationale: [ADR-0201](adr/0201-crd-model.md). diff --git a/docs/roadmap/planned-features.md b/docs/roadmap/planned-features.md index 0eeb4525..b6a52880 100644 --- a/docs/roadmap/planned-features.md +++ b/docs/roadmap/planned-features.md @@ -106,9 +106,10 @@ phased ⬜ backlog, and items that need a design pass before implementation. For | | | | --- | --- | -| **Status** | Planned | -| **Summary** | Wire **`KollectConnectionTest`** and sink annotation probes for **`gcs`** and **`nats`** sink types (Git, Postgres, Kafka, S3, GitLab probes exist; GCS/NATS marked ⬜ in CR reference). | -| **Related ADRs** | [ADR-0403](../adr/0403-connection-test.md) · [kollectsink CR reference](../crds/kollectsink.md) | +| **Status** | Shipped (Phase 1) | +| **Roadmap** | Phase 1 ✅ | +| **Summary** | **`KollectConnectionTest`** and family sink annotation probes for **`gcs`** (`KollectSnapshotSink`) and **`nats`** (`KollectEventSink`) — alongside Git, Postgres, Kafka, S3, and GitLab probes. | +| **Related ADRs** | [ADR-0403](../adr/0403-connection-test.md) · [ADR-0414](../adr/0414-sink-family-crds.md) · [KollectSnapshotSink](../crds/kollectsnapshotsink.md) · [KollectEventSink](../crds/kollecteventsink.md) | --- diff --git a/test/e2e/fixtures/multitenant/tenant-inventory.yaml.template b/test/e2e/fixtures/multitenant/tenant-inventory.yaml.template index 57ccb639..274a4ddf 100644 --- a/test/e2e/fixtures/multitenant/tenant-inventory.yaml.template +++ b/test/e2e/fixtures/multitenant/tenant-inventory.yaml.template @@ -4,5 +4,4 @@ metadata: name: tenant-inventory namespace: ${TENANT_NS} spec: - sinkRefs: [] suspend: false diff --git a/test/e2e/fixtures/multitenant/tenant-scope.yaml b/test/e2e/fixtures/multitenant/tenant-scope.yaml index 80cb568b..bdd6f53f 100644 --- a/test/e2e/fixtures/multitenant/tenant-scope.yaml +++ b/test/e2e/fixtures/multitenant/tenant-scope.yaml @@ -10,5 +10,3 @@ spec: kind: Deployment allowedNamespaces: - kollect-tenant-a - sinkRefs: - - git-inventory-demo