sync to refactored break - #2
Open
brownzebra wants to merge 2392 commits into
Open
Conversation
merge main into grafana-user-and-httproute
Grafana user and httproute, keeping old dashboards, premade kubernetes dashboards, initial cluster health dashboard
get grafana into v210 branch
Revert "get grafana into v210 branch"
…ubernetes dashboards, initial cluster health dashboard"
…route Revert "Grafana user and httproute, keeping old dashboards, premade kubernetes dashboards, initial cluster health dashboard"
Revert "Revert "get grafana into v210 branch""
Polling (oci://):
apps:
aiwb:
repoURL: oci://ghcr.io/silogen/charts/aiwb
repoVersion: "0.0.0-int-test"
# path defaults to "."
Non-polling (Helm OCI):
apps:
aiwb:
repoURL: ghcr.io/silogen/charts # bare hostname
chart: aiwb
repoVersion: "0.0.0-int-test"
…y and node labeling
… cluster managers
… child app repo URLs
…eycloak running earlier
Details to deploy on OpenShift environment
Provision a write credential for AIWB's ADR-0002 Scenario 2 key store, scoped to secrets/apikeys/* instead of reusing the root token: - secret-manager (reconciling, every 5m + init) idempotently ensures apikeys-write-policy and mints a scoped token at secrets/aiwb-openbao-token, re-minting only when the stored one is missing or invalid. No renewal machinery (nothing else in the stack renews); the token carries the default TTL and rotates on expiry. - aiwb-external-secrets: an ExternalSecret materializes that token into the aiwb namespace as aiwb-openbao-token, which the AIWB backend already consumes as OPENBAO_TOKEN. ESO reads it via the existing readonly store. Policy capabilities match the AIWB client exactly: data create/read/update and metadata create/update/delete (delete_version_after + revocation). The OCI publish of aiwb-external-secrets and the root repoVersion bump are handled by the external release pipeline, not here.
docs: update 2.2.0 refs to 2.2.2 in migration from 2.1.x guide
…edential EAI-7277: scoped OpenBao write credential for the API-key KV path
Add option to use local Helm values for apps
Minor correction
A feature which allows addition of extra apps managed by argo.
* feat: overhaul LGTM stack v1.0.8 with official Grafana image * lgtm overhaul v4 * old PROMETHEUS_EXTRA_ARGS to single line so retention.size flag isn't dropped * expose collector memory_limiter settings via values.yaml * docs: add otel-lgtm-stack per-cluster tuning guide * docs: polishing the doc
* Upgrade seaweedfs operator to 0.1.29 * Disable seaweed ingress, as we create HTTPRoutes. Remove seaweed crds (part of operator now) * Replace init-job with seaweedfs native bucket yaml definition * fix wrong values object reference * Add missing config to root values * Add missing value from root values * Fix scoping error in the bucket template * Upgrade to latest (0.1.36) * fix wrong value for Versioning attribute * Fix windows end line issue * Add quotation marks to get helm to correctly read value as string * Change the backwards-compatibility-enabling service "filer-s3" (the naming is anathema to the latest seaweed operator) to point to the new seaweedfs s3 service * Keep the httproute unchanged, no unneeded changes should be added * add a dummy file in crds, to allow migrating without removing seaweedfs-crds from apps * Update SBOM * update sbom components.yaml
A CloudNativePG Cluster with instances: 1 gets a <name>-primary PodDisruptionBudget at minAvailable: 1, which permits zero disruptions for as long as the cluster is single-instance. That PDB protects nothing, since there is no second replica to keep serving, and it blocks kubectl drain on whichever node the pod sits on. Node maintenance then stalls until an operator disables it by hand on the live cluster, out of band from Git. enablePDB now defaults to false while instances is 1 and true from two instances up, and an explicit cnpg.enablePDB wins either way.
keycloak-config renders the same keycloak-cnpg Cluster as keycloak-old, so it carries the same drain blocker. A single-instance PDB permits zero disruptions, which protects nothing and only stalls `kubectl drain` on whichever node the primary sits on. Introduce the cnpg values block this chart lacked and derive enablePDB from instances, with an explicit value winning. Matches the pattern already applied to keycloak-old in this branch.
EAI-7854 Skip the keycloak CNPG PDB for a single instance
The small profile had no override for aiwb-infra-cnpg, so it inherited instances: 3 from the published aiwb-cnpg chart, while the medium profile explicitly set instances: 1. A small cluster therefore ran three Postgres replicas silently, consuming more resources than a medium cluster. airm-infra-cnpg already defaults to a single instance upstream, so it needs no override. The large profile keeps three instances.
EAI-7854 Pin aiwb-infra-cnpg to one instance on small profile
…G topology Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
EAI-7889 Add pod anti-affinity for SeaweedFS and fix keycloak-old CNPG topology
The airm-realm.json value is a YAML block scalar with 4-space indented content. The closing bracket of the emptied KeyProvider array sat at column 0, which terminates the block scalar early, so the parser read it as a new mapping key. This made 'helm template sources/keycloak-config' fail with 'line 2123: did not find expected key'. Indenting the bracket restores rendering. The embedded JSON is byte-identical after re-parsing, the empty array was intentional.
feat: extraApps for argo
EAI-7854 Repair YAML block scalar in keycloak realm ConfigMap
Minor fixes for Openshift Radeon clusters
Improving script and reducing required files
…auth (#794) * feat: bump versions of envoy and envoy-ai-gateway * feat: bump versions of envoy and envoy-ai-gateway at root/values.yaml * Restore envoy-gateway RBAC for AI Gateway CRDs The v0.6.0 -> v1.0.0 envoy-ai-gateway chart upgrade dropped the ClusterRole/ClusterRoleBinding granting the envoy-gateway service account get/list/watch on aigateway.envoyproxy.io aigatewayroutes and aiservicebackends. These are configured as extensionManager resources, so envoy-gateway watches them directly and was denied, causing its informer caches to never sync and the controller to crashloop (0/1). Re-add the template to close the RBAC gap. * Pin ai-gateway controller/extProc to build with #2304 v1.0.0's controller fails its gateway reconcile with "found gateway-labeled objects in multiple namespaces" whenever the Gateway lives in the Envoy Gateway namespace (envoy-gateway-system), because getObjectsForGateway does not dedup gw.Namespace against envoyGatewayNamespace. This blocks extproc filter-config generation, so the ai-gateway proxy cannot roll to v1.0.0. Fixed upstream in envoyproxy/ai-gateway#2304 (not yet in a tagged release). Pin controller and extProc images to a main build containing the fix (v1.0.0 + 11 commits). Remove once a release past v1.0.0 ships #2304. * Bring ai-gateway native-auth cutover into the upgrade branch (EAI-7304) bump_version had the envoy/ai-gateway version bump but not the ai-gateway auth cutover, so ai-gateway was still behind cluster-auth ext_authz — which does not reject the header-less/no-key path, leaving the API-key bypass open (verified live on app-dev: no/bogus key with no x-ai-eg-* headers -> 200). Port the EAI-7304 changes from EAI-7502 (final, post-revert state): - security-policy-ai-gateway-default-deny.yaml: gateway-scoped defaultAction Deny backstop on ai-gateway; per-model apiKeyAuth policies from ai-gateway-discovery override it per-route, so keyless/unmatched requests are denied instead of served. - security-policy-extauth.yaml: drop ai-gateway from cluster-auth ext_authz (two gateway-scoped SecurityPolicies cannot target the same Gateway). - values.yaml: document the cutover (gated on aiGateway.enabled). - ai-gateway-proxy-config.yaml / envoy-proxy-access-logs.yaml: attribute aim_service_id via x-ai-eg-backend. * ai-gateway: follow POC — lua before api_key_auth + clearRouteCache Test the POC's mechanism for closing the header-less API-key bypass: - filterOrder: move lua before api_key_auth (was before ext_authz, a no-op since ext_authz was removed from ai-gateway). - set-model-header-from-body Lua: call clearRouteCache() after setting x-ai-eg-backend/model so the request re-selects the keyed HTTPRoute before api_key_auth evaluates. Empirical trial: cluster-forge routes on backend-UUID (ext_proc-resolved), unlike the POC's model-name routing, so this may not fully close it — testing on app-dev to confirm. * Revert "ai-gateway: follow POC — lua before api_key_auth + clearRouteCache" This reverts commit 5d93cd0. * ai-gateway: lua before api_key_auth + clearRouteCache (re-test) Reinstates the reverted POC to close the header-less API-key bypass on the upgraded stack (EG v1.8.1 / AI Gateway v1.0.0). filterOrder moves lua ahead of api_key_auth so set-model-header-from-body derives x-ai-eg-model/backend from the body and clearRouteCache() re-selects the keyed HTTPRoute before per-model apiKeyAuth evaluates. Discovery still emits the SP-covered model-name rule on app-dev, so a model-only body (no backend UUID) lands on an auth-covered route -- the condition that was absent when this was first reverted. To validate on app-dev: confirm lua relocates to filter index 0 (config_dump) and the body-only no/bogus-key path returns 401. * Revert lua-before-api_key_auth POC (confirmed ineffective) Re-tested live on app-dev (EG v1.8.1 / AI Gateway v1.0.0): the reorder was honored (lua moved ahead of api_key_auth) but the body-only bypass stayed open. Root cause proven via access logs -- ext_proc/aigateway anchors ahead of lua (lua is in afterExtProcFilterPrefixes) and consumes the body, so the Lua's request_handle:body() is empty and it never sets x-ai-eg-model. Config filterOrder cannot place lua ahead of ext_proc. Reverting to baseline before the EnvoyPatchPolicy-based approach (json_to_metadata + header_mutation ahead of api_key_auth, which are NOT in the anchoring prefix list). * ai-gateway: EnvoyPatchPolicy to set x-ai-eg-model from body before auth Variant A of the header-less API-key bypass fix. Enables enableEnvoyPatchPolicy and injects json_to_metadata + header_mutation at the front of the ai-gateway listener, ahead of api_key_auth. These filters are not in the ai-gateway afterExtProcFilterPrefixes anchor list, so (unlike the reverted Lua POC) they run before ext_proc consumes the body; json_to_metadata reads the body non-destructively so ext_proc still gets it. x-ai-eg-model is set from the body model field before api_key_auth, so a body-only request matches the SP-covered model route and authenticates. Requires an envoy-gateway restart after sync (enableEnvoyPatchPolicy). Open item to validate on app-dev: route re-computation -- if the catch-all route is cached before header_mutation runs, api_key_auth may read stale per-route config; the fallback is Variant B (buffer exception + Lua clearRouteCache). * ai-gateway: clearRouteCache before api_key_auth so the SP-covered route wins Variant A set x-ai-eg-model from the body before api_key_auth, but the route was still the cached catch-all (header_mutation does not clear the route cache), so api_key_auth passed the SP-less route and the body-only bypass stayed 200. Add a tiny clearRouteCache Lua between header_mutation and api_key_auth so api_key_auth re-resolves the route from the now-set x-ai-eg-model header (the SP-covered model route). Named non-prefixed so ai-gateway does not anchor its ext_proc ahead of it. * ai-gateway: set x-ai-eg-model in Lua from metadata (header_mutation didn't apply) Live stats showed json_to_metadata.rq.success (body parsed, metadata set) but api_key_auth.allowed on body-only requests -- i.e. header_mutation's %DYNAMIC_METADATA% value never populated x-ai-eg-model, so api_key_auth still saw the catch-all route. Replace header_mutation with a custom-named Lua that reads ai.body:model from dynamic metadata, sets x-ai-eg-model when absent, and clearRouteCache() so api_key_auth re-resolves onto the SP-covered model route. * ai-gateway: read body directly in Lua (metadata was set too late, wrong phase) json_to_metadata populates dynamic metadata in the data phase, but the Lua and api_key_auth run in the header phase -- so the metadata was never available when the Lua read it, and x-ai-eg-model stayed unset. Switch to a single Lua that reads the body directly (request_handle:body() blocks until the body is buffered, so it works in the header phase). The Lua is ahead of ext_proc (non-prefixed name) so the body is intact; it sets x-ai-eg-model when absent and clearRouteCache so api_key_auth re-resolves onto the SP-covered model route. * ai-gateway: temp debug - report Lua body/model result via x-lua-debug header * Revert EnvoyPatchPolicy experiments; restore lua-before-api_key_auth POC The EnvoyPatchPolicy path (json_to_metadata/header_mutation/body-read Lua) was proven a dead end: the body->header transform works (x-ai-eg-model set before api_key_auth, confirmed) but api_key_auth does not honor the per-model SP on a clearRouteCache-re-resolved route, so the body-only bypass stayed 200. Restore the branch to the POC state (filterOrder lua-before-api_key_auth + clearRouteCache) and drop enableEnvoyPatchPolicy + the EnvoyPatchPolicy manifest. * ai-gateway: wire body-aware ext_authz on the default-deny SecurityPolicy Merge an extAuth block into the gateway-scoped ai-gateway-default-deny policy (Envoy Gateway allows only one SecurityPolicy per Gateway) so header-less inference requests -- standard OpenAI clients that put the model in the body -- are authorized by ai-gateway-discovery's ext_authz handler. bodyToExtAuth forwards the body, Authorization is forwarded in, and x-api-key-id back out for metrics. Add the cross-namespace ReferenceGrant for the discovery Service. On keyed HTTPRoutes the per-model route SecurityPolicy overrides this whole policy (extAuth included), so extAuth only authorizes the header-less path -- a supplement to native per-model apiKeyAuth until upstream ai-gateway supports body-path client auth. Requires the discovery image/chart that exposes the authz port. * ai-gateway: point body ext_authz backendRef at the discovery namespace The ai-gateway-discovery Service is deployed in the ai-gateway-discovery namespace (ArgoCD destination), not ai-gateway-system. Fix the extAuth backendRef and the ReferenceGrant to target ai-gateway-discovery so the SecurityPolicy is accepted. * Revert to ai-gateway-system: canonical discovery namespace b078992 changed the ext_authz backendRef/grant to ai-gateway-discovery to match what was live on app-dev, but that deployment is a test install straight from the core branch (github silogen/core, chart into ai-gateway-discovery), not the canonical path. cluster-forge declares the discovery app in ai-gateway-system (published chart), so the extAuth must target ai-gateway-system. * ai-gateway: parameterize ext_authz discovery namespace Hardcoding the discovery namespace was fragile: it's ai-gateway-system canonically but differs per env (app-dev's overlay runs discovery in ai-gateway-discovery), and a mismatch makes the SecurityPolicy fail to attach (extAuth silently not applied -> the body-only bypass returns). Drive the extAuth backendRef + ReferenceGrant namespace from .Values.aiGateway.discoveryNamespace, wired in root/values.yaml to .Values.apps.ai-gateway-discovery.namespace so it always matches where discovery deploys. * fix: use index for the dashed app key in the discoveryNamespace helmParameter Go templates can't dot-access a map key containing '-', so {{ .Values.apps.ai-gateway-discovery.namespace }} failed helm lint. Use {{ index .Values.apps "ai-gateway-discovery" "namespace" }}. * fix: restore ai-gateway metric header attributes The v1.0.0 ai-gateway chart ships requestHeaderAttributes and metricsRequestHeaderAttributes as empty strings (v0.6.0 populated them), so the deployment template omits the --metricsRequestHeaderAttributes controller flag. Without it the ext_proc never maps x-api-key-id -> api_key_id (nor x-ai-eg-backend -> aim_service_id) onto the gen_ai_* metrics, leaving those labels empty and breaking per-API-key usage stats in the AIWB UI. Set both attributes in the envoy-ai-gateway valuesObject. aim_service_id tracks x-ai-eg-backend to match the access-log mapping, since the new gateway build no longer emits x-aim-service-id. * fix: set ai-gateway header attributes in the chart default Move the requestHeaderAttributes / metricsRequestHeaderAttributes mapping from the root valuesObject override into the vendored v1.0.0 chart default, matching how the v0.6.0 chart shipped them. The v1.0.0 chart defaulted them to "", which dropped the controller flags and blanked api_key_id / aim_service_id on the gen_ai_* metrics — breaking AIWB per-API-key usage stats for any install, not just this one. Keeping it in the chart default (single source of truth) rather than a root override means fresh installs and other environments get correct per-key metrics without extra wiring. aim_service_id maps from x-ai-eg-backend since the current build no longer emits x-aim-service-id. The ai-gateway controller/extProc image stays pinned to 8007dfd (v1.0.0 + 11 commits) because the crash fix #2304 is not yet in any release; v1.0.0 is the latest and reintroduces the reconcile crash. Revisit when a release ships past v1.0.0. * chore: remove dead code from superseded ai-gateway approaches Clean up the branch for review now that the final design has settled (native apiKeyAuth cutover + PR 793 body-aware ext_authz via discovery): - Remove the set-model-header-from-body Lua EnvoyExtensionPolicy and its filterOrder wiring on the ai-gateway EnvoyProxy. It was a superseded POC (repeatedly reverted as ineffective): header-less/body requests are authorized by ai-gateway-discovery's body-aware ext_authz, which parses backend/model from the forwarded body and enforces per-backend/per-model auth. Verified on app-dev that the Lua is not the effective mechanism (wrong-backend requests return discovery's error, not native apiKeyAuth's). - Drop the old vendored charts left by the version bump (envoy-gateway/v1.7.1, envoy-ai-gateway/v0.6.0, envoy-ai-gateway-crds/v0.6.0), now unreferenced by root/values.yaml, and point sbom/components.yaml at the current versions. - Refresh stale comments that referenced the removed lua-before-ext_authz chain. No behavior change intended; the apps-gateway (https) cluster-auth filterOrder is unrelated and left as-is. * EAI-7243: forward x-ai-eg-backend for header-less metric attribution Add x-ai-eg-backend to the default-deny SP's headersToBackend so the resolved backend that discovery forwards (core #4309) reaches the ext_proc and populates aim_service_id on the header-less path, where the gateway can't resolve the backend from EDS endpoint metadata. Stacked on the bump branch (that's where the default-deny SP lives); retarget to main after #794 merges. * EAI-7864: Sanitize non-UUID x-ai-eg-backend before it is recorded as aim_service_id Add a sanitization-only Lua EnvoyExtensionPolicy (sanitize-backend-header) that drops x-ai-eg-backend unless it is a well-formed workload UUID, so a client- supplied bogus value (e.g. "foobar") is never recorded as the aim_service_id access-log field / gen_ai metric label. Distinct from the removed set-model-header-from-body POC: it does not read the body, set model/backend, route, clear the route cache, or authorize — pure metric hygiene. Removing a non-UUID value is safe for routing (such a value can never match a backend route rule's Exact UUID match). Re-adds a minimal filterOrder (lua before api_key_auth and ext_proc) so the strip precedes header capture; metric coverage is ordering-dependent and must be verified on deploy. Real cross-model attempts (a valid foreign backend UUID) are already denied by native apiKeyAuth via the EAI-7864 routing fix; a valid-format fake UUID is an accepted residual. --------- Co-authored-by: woojae-siloai <woojae.lim@amd.com> Co-authored-by: lauri-amd <lauri.nuutinen@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.