diff --git a/docs/openshift/README.md b/docs/openshift/README.md index dd7cf4dc..ffbfa75b 100644 --- a/docs/openshift/README.md +++ b/docs/openshift/README.md @@ -74,7 +74,7 @@ The `install.sh` script runs through the numbered phases below. Each phase print | Step | Phase | Description | |------|-------|-------------| | 1 | Downloading cluster-forge release | Downloads and extracts the pinned cluster-forge release tarball (`CLUSTER_FORGE_VERSION`), fetches `manual_helm_install` secrets and scripts from GitHub, and applies post-clone patches (e.g. envoy-gateway SecurityPolicy `failOpen: true`). | -| 2 | Custom SecurityContextConstraints (SCCs) | Applies OpenShift custom SCC manifests from `extra/scc.yaml` so pods with non-default security contexts (e.g. OpenTelemetry operator with `seccompProfile: RuntimeDefault`) can be scheduled. | +| 2 | Custom SecurityContextConstraints (SCCs) | Applies OpenShift custom SCC manifests from `extra/01-scc.yaml` so pods with non-default security contexts (e.g. OpenTelemetry operator with `seccompProfile: RuntimeDefault`) can be scheduled. | | 3 | local-path provisioner & default StorageClass | Ensures dynamic storage is available: installs the local-path provisioner if missing, creates a `default` StorageClass when absent, and marks it as the cluster default. | | 4 | Kuberay operator | Installs the Kuberay operator for Ray cluster management. | | 5 | CloudNativePG database operator | Installs the CloudNativePG (CNPG) operator in `cnpg-system` and waits for it to be ready. Skipped when `PLUGGABLE_DB=true`. | @@ -93,7 +93,7 @@ The `install.sh` script runs through the numbered phases below. Each phase print | 18 | KEDA (event-driven autoscaling) | Installs KEDA operator, metrics server, and admission webhooks. | | 19 | Kedify OTEL scaler | Installs the Kedify OpenTelemetry metrics scaler for KEDA. | | 20 | MetalLB configuration (skipped on OpenShift) | Skipped; OpenShift provides its own load balancer and routing. | -| 21 | Envoy Gateway (skipped on OpenShift; creating namespaces) | Skips Envoy Gateway installation; creates `envoy-gateway-system` and `cluster-auth` namespaces referenced by other components. OpenShift Routes (via Kyverno) replace Gateway API routing. | +| 21 | Envoy AI Gateway | Serves `https://ai.`, the single endpoint every deployed model answers on, with the model chosen per request from the `x-ai-eg-backend` and `x-ai-eg-model` headers. Ordinary web traffic is unaffected and keeps using OpenShift Routes generated by Kyverno; only inference needs this, because a Route cannot match on headers. Installs the Envoy Gateway and AI Gateway control planes, the OpenShift-specific SCC/RBAC/Route from `extra/06-ai-gateway.yaml`, and a copy of the router's wildcard certificate as `cluster-tls`. Skipped on clusters that do not serve `route.openshift.io`. See "AI gateway" below. | | 22 | KServe (model serving) | Installs KServe CRDs and operator in RawDeployment mode, or skips if KServe is already running (e.g. under RHOAI in `redhat-ods-applications`). | | 23 | AMD GPU operator (NFD + KMM + device plugin) | Installs the AMD GPU Operator (Node Feature Discovery, Kernel Module Management, device plugin) if not already present. | | 24 | AMD GPU operator config (DeviceConfig) | Applies a DeviceConfig in the operator namespace when no DeviceConfig exists anywhere in the cluster. | @@ -111,5 +111,122 @@ The `install.sh` script runs through the numbered phases below. Each phase print | 36 | Kueue | Installs the Kueue job queueing controller and applies cluster configuration from `kueue-config`. | | 37 | Kaiwo CRDs | Installs Kaiwo custom resource definitions in `kaiwo-system`. | | 38 | Kaiwo | Installs the Kaiwo operator and applies `kaiwo-config` manifests (with ExternalSecret API version fixes). | -| 39 | Apply OpenShift Routes (AIWB UI/API + Keycloak) | Applies native OpenShift Routes from `extra/routes.yaml` for AIWB UI, API, and Keycloak, substituting the cluster domain. | +| 39 | Apply OpenShift Routes (AIWB UI/API + Keycloak) | Applies native OpenShift Routes from `extra/09-routes.yaml` for AIWB UI, API, and Keycloak, substituting the cluster domain. | | 40 | Cleanup (remove downloaded cluster-forge sources) | Removes the downloaded cluster-forge release directory from `CLUSTER_FORGE_DIR` to free disk space. | + +### The `extra/` manifests + +Everything `install.sh` applies that is not rendered from a cluster-forge chart lives in `extra/`. The numeric prefix is the order the script applies them in, so reading the directory top to bottom tells you what lands on the cluster and when, without grepping the script. + +| File | Applied in | What it is | +|------|-----------|------------| +| `01-scc.yaml` | 2 | Every custom SCC, the `anyuid` grants, the Kyverno SCC-generator RBAC, and the aim-system SCCs for the detector DaemonSets and discovery Jobs | +| `02-local-path-provisioner-scc.yaml` | 3 | Lets the local-path provisioner mount hostPath | +| `03-local-path-helper-pod-selinux.yaml` | 3 | Merge-patch, not a manifest: runs the helper pod as `spc_t` so it can mkdir under `/var/opt` | +| `04-local-path-access-mode-scoped.yaml` | 8 | Scopes cluster-forge's RWX→RWO mutation to the local-path StorageClasses | +| `05-kyverno.yaml` | 9 | Per-project SCC generation, HTTPRoute→Route conversion, orphaned-SCC cleanup, and the RBAC each needs | +| `06-ai-gateway.yaml` | 21 | The AI gateway's SCC, RBAC and passthrough Route | +| `07-ai-gateway-values.yaml` | 21 | Helm values for the `envoy-gateway` chart, not a manifest | +| `08-amd-gpu-nodefeaturerule.yaml` | 25 | NodeFeatureRule labelling AMD GPU nodes | +| `09-routes.yaml` | 39 | Routes for the AIWB UI/API and Keycloak | + +Order is load-bearing in two places. Across files, RBAC and SCCs must exist before the workloads they admit. Within `05-kyverno.yaml`, each RBAC ClusterRole must precede the policy that depends on it, because Kyverno's webhooks validate those permissions at admission time and reject the policy otherwise. + +When `install.sh` is run from a checkout it reads these files from the `extra/` directory beside it. Piped straight from `curl` it has no directory, so it fetches each one from `docs/openshift/extra/` on cluster-forge `main` and aborts with the filename and URL if one is missing. + +That only works because the two directories hold the same files under the same names, prefix included. Renaming a file here without renaming it upstream, or the other way round, breaks the piped install for everyone while leaving checkout runs working — so it fails for whoever is least able to debug it. Rename in both places or in neither. + +--- + +5. Chart versions + +Two different versions are in play and they are easy to confuse. + +`CLUSTER_FORGE_VERSION` selects which release tarball is downloaded. It does **not** select chart versions: the tarball is a catalogue that carries several versions of the same chart side by side (for example `sources/kserve` holds both `v0.15.2` and `v0.16.0`). + +Which one gets installed is declared by the release itself, in `root/values.yaml`, as `apps..path` for charts that live in the tarball and `apps..repoVersion` for the ones pulled from an OCI registry. The OCI charts — AIWB, AIM Engine, Kaiwo, the AI gateway discovery controller — are not in the tarball at all, and their numbering is unrelated to the cluster-forge release: at v2.2.2 AIWB is 2.0.0, AIM Engine is 0.2.5 and Kaiwo is v0.2.1. There is no `2.2.2` tag for any of them, so a chart version cannot be derived from the release number. + +`install.sh` reads `repoVersion` out of the downloaded release for every OCI chart, so bumping `CLUSTER_FORGE_VERSION` picks up whatever that release pins, with nothing to edit by hand. + +To install a chart that no release references yet, override it per app. The variable is the app key uppercased with dashes turned into underscores: + +```bash +CF_VERSION_AIWB=2.0.1 CF_VERSION_AI_GATEWAY_DISCOVERY=2.0.1 ./install.sh +``` + +The override is announced in the output, since it puts the cluster on a combination the release was not tested with. + +Charts that come from the tarball still have their version written into the path in `install.sh`. In every case that is the newest version the tarball holds, with one deliberate exception: `external-secrets` is installed at `0.19.2` while v2.2.2 declares `0.15.1`. Keep that in mind before making those dynamic too — doing it naively would downgrade it. + +--- + +6. AI gateway + +Every model deployed on the cluster is served from one hostname, `https://ai.`. The model is chosen per request from two headers rather than from the URL: + +```bash +curl -k -X POST https://ai./v1/chat/completions \ + -H 'Content-Type: application/json' \ + -H 'x-ai-eg-backend: ' \ + -H 'x-ai-eg-model: ' \ + -d '{"messages":[{"role":"user","content":"Hello"}],"stream":false}' +``` + +The workload UUID is the `airm.silogen.ai/workload-id` label on the InferenceService: + +```bash +kubectl get inferenceservice -A -o custom-columns=\ + 'NAME:.metadata.name,UUID:.metadata.labels.airm\.silogen\.ai/workload-id' +``` + +Nothing has to be configured per model. The AI Gateway Discovery controller (step 34) watches InferenceServices and generates the AIGatewayRoute for each one as it becomes ready. + +This is the one place where OpenShift's own routing is not enough: a Route matches on host and path only, so a set of models sharing a hostname cannot be told apart by it. Everything else on the cluster keeps using Routes generated from HTTPRoutes by Kyverno, and models stay reachable on their individual `workloads.` URLs in parallel. + +The step needs an OpenShift Route to be reachable at all and an OpenShift SCC to get its pods admitted, so it runs only where `route.openshift.io` is served. Anywhere else it is skipped with a message and nothing is installed. + +### How this differs from the RKE2 reference clusters + +Those clusters chain two Envoy gateways on a single `:443` VIP. A front-door `https` Gateway owns the whole domain and hands AI traffic down by SNI, without decrypting it: + +``` +*. -> Gateway "https" (LoadBalancer front door) + |- listener https TLS terminate -> apps + |- listener k8s-passthrough TLS passthrough -> kube API + '- listener ai-passthrough TLS passthrough -> TLSRoute + | + Gateway "ai-gateway" (ClusterIP, own Envoy) <---' + terminates cluster-tls, ext_proc dispatch, + InferencePool -> EPP -> model pod +``` + +On OpenShift the HAProxy router already **is** that front door, so the `https` Gateway is deliberately not installed — installing a second front door is exactly what would collide with cluster ingress. Only the second tier is added, and the TLSRoute is replaced by one passthrough Route: + +``` +ai.:443 -> HAProxy router (passthrough by SNI, terminates nothing) + -> Service ai-gateway:443 + -> Gateway ai-gateway (terminates cluster-tls) + -> ext_proc x-ai-eg-backend + -> InferencePool -> EPP -> model pod +``` + +Everything below the passthrough point is identical on both, which is why the same cluster-forge chart serves both with only three patched lines (exact listener hostname, no `cluster-bloom/first-node` nodeSelector, and a capped Envoy `concurrency`). + +The Route is TLS-passthrough, so HAProxy presents no certificate of its own and Envoy terminates TLS with the `cluster-tls` secret in `envoy-gateway-system`. The installer populates it by **copying** the wildcard certificate the router already serves. It is a copy, not a reference, so it goes stale when the ingress-operator rotates the original — the symptom is a TLS error on `ai.` alone, with every other hostname fine. + +Re-running `install.sh` refreshes it, and so does copying the secret by hand if a full re-run is not wanted: + +```bash +kubectl get secret router-certs-default -n openshift-ingress -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/tls.crt +kubectl get secret router-certs-default -n openshift-ingress -o jsonpath='{.data.tls\.key}' | base64 -d > /tmp/tls.key +kubectl create secret tls cluster-tls -n envoy-gateway-system \ + --cert=/tmp/tls.crt --key=/tmp/tls.key --dry-run=client -o yaml \ + | kubectl apply --server-side --force-conflicts -f - +rm -f /tmp/tls.crt /tmp/tls.key +``` + +(If the IngressController has a `spec.defaultCertificate`, use that secret name instead of `router-certs-default`.) + +`-k` is needed for as long as the router serves the cluster's default self-signed certificate. Install a CA-signed certificate on the IngressController and re-run the script above, and it applies to this endpoint too. + +Requests are not authenticated yet. The gateway ignores the `Authorization` header because ext_authz is not enabled: the cluster-auth shim (step 28) speaks REST on 8081, while the SecurityPolicy the chart ships expects gRPC on 50051. diff --git a/docs/openshift/extra/scc.yaml b/docs/openshift/extra/01-scc.yaml similarity index 81% rename from docs/openshift/extra/scc.yaml rename to docs/openshift/extra/01-scc.yaml index 18a45862..f90edbc9 100644 --- a/docs/openshift/extra/scc.yaml +++ b/docs/openshift/extra/01-scc.yaml @@ -1,3 +1,19 @@ +# ============================================================================= +# Custom SecurityContextConstraints for OpenShift +# ============================================================================= +# Every SCC install.sh applies in STEP 1, plus the RBAC that goes with them: +# the per-namespace `anyuid` grants and the ClusterRole that lets Kyverno +# generate per-project SCCs. +# +# The last section holds the aim-system SCCs. They are separate objects rather +# than extra `users:` entries on the ones above because the aim-engine detector +# DaemonSets and discovery Jobs need hostPath and privileged, which `anyuid` +# does not allow. +# +# This file supersedes the separate scc.yaml and scc-aim-system.yaml that +# cluster-forge main carried until 2026-08-13. Both were deleted in the same +# change that added this one; a piped install.sh fetches this file by name, so +# do not restore them. --- apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints @@ -433,7 +449,7 @@ users: # KYVERNO SCC-GENERATOR RBAC # ============================================================================ # Lets the Kyverno controllers manage SecurityContextConstraints so the -# generate-scc-on-namespaces ClusterPolicy (kyverno-scc-for-ns.yaml) can create +# generate-scc-on-namespaces ClusterPolicy (05-kyverno.yaml) can create # per-project SCCs. The background-controller performs the generate; the # admission-controller needs list/get or the policy validation webhook rejects # the ClusterPolicy at apply time. Safe to exist before Kyverno is installed. @@ -866,3 +882,123 @@ subjects: - kind: ServiceAccount name: kaiwo-kaiwo-operator-chart-controller-manager namespace: kaiwo-system + +# ============================================================================= +# aim-system: SCCs for the aim-engine detector DaemonSets and discovery Jobs +# ============================================================================= +# Extra SecurityContextConstraints for the aim-system namespace. +# +# scc.yaml already grants anyuid to system:serviceaccounts:aim-system +# (ClusterRoleBinding anyuid-scc-aim-system). anyuid allows an arbitrary UID but +# NOT hostPath volumes, NOT privileged containers and NOT seccomp profiles, so +# the aim-engine accelerator-detector DaemonSets and the model discovery Jobs +# are rejected at admission and never create a pod at all: +# +# daemonset/...-accelerator-detector-cpu DESIRED 1 CURRENT 0 (466 x FailedCreate) +# daemonset/...-accelerator-detector-gpu DESIRED 1 CURRENT 0 (464 x FailedCreate) +# Error creating: pods "..." is forbidden: unable to validate against any +# security context constraint: ... hostPath volumes are not allowed to be used +# +# Both SCCs below are bound to named service accounts via users:, never to a +# group, so no other workload in the cluster can select them. +# +# priority is deliberately left unset. SCC selection then prefers the most +# restrictive constraint that can admit the pod, so these are only reached after +# restricted-v2 and anyuid have already failed to validate. +--- +apiVersion: security.openshift.io/v1 +kind: SecurityContextConstraints +metadata: + name: cf-aim-engine-detector-scc + annotations: + kubernetes.io/description: >- + Lets the aim-engine accelerator-detector DaemonSets mount host paths and, + for the GPU variant, run privileged. Bound to one service account. +# Required by both DaemonSets: +# /etc/kubernetes/node-feature-discovery/features.d/ writes NFD feature files +# /proc/cpuinfo cpu variant +# /dev/kfd, /dev/dri gpu variant +allowHostDirVolumePlugin: true +allowHostIPC: false +allowHostNetwork: false +allowHostPID: false +allowHostPorts: false +# gpu variant sets containers[0].securityContext.privileged: true so it can +# reach /dev/kfd and /dev/dri +allowPrivilegedContainer: true +allowPrivilegeEscalation: true +# Neither pod adds a capability: the cpu detector drops ALL itself and the gpu +# detector relies on privileged, so nothing needs to be granted or forced. +allowedCapabilities: [] +defaultAddCapabilities: [] +requiredDropCapabilities: [] +readOnlyRootFilesystem: false +# initContainer prep-features-dir runs as 0, detector container as 65534 +runAsUser: + type: RunAsAny +seLinuxContext: + type: RunAsAny +fsGroup: + type: RunAsAny +supplementalGroups: + type: RunAsAny +seccompProfiles: + - '*' +userNamespaceLevel: AllowHostLevel +volumes: + - configMap + - downwardAPI + - emptyDir + - hostPath + - persistentVolumeClaim + - projected + - secret +groups: [] +users: + - system:serviceaccount:aim-system:aim-engine-aim-engine-chart-accelerator-detector +--- +apiVersion: security.openshift.io/v1 +kind: SecurityContextConstraints +metadata: + name: cf-aim-engine-discovery-scc + annotations: + kubernetes.io/description: >- + Lets the aim-engine model discovery Jobs set a seccomp profile and run as + UID 65532. No host access of any kind. +allowHostDirVolumePlugin: false +allowHostIPC: false +allowHostNetwork: false +allowHostPID: false +allowHostPorts: false +allowPrivilegedContainer: false +allowPrivilegeEscalation: false +allowedCapabilities: [] +defaultAddCapabilities: [] +requiredDropCapabilities: + - ALL +readOnlyRootFilesystem: false +# pods set runAsUser 65532, outside the namespace-allocated UID range +runAsUser: + type: RunAsAny +seLinuxContext: + type: RunAsAny +fsGroup: + type: RunAsAny +supplementalGroups: + type: RunAsAny +# pods set securityContext.seccompProfile.type: RuntimeDefault, which anyuid +# rejects with "metadata.annotations[seccomp...]: Forbidden: seccomp may not be set" +seccompProfiles: + - runtime/default +userNamespaceLevel: AllowHostLevel +volumes: + - configMap + - downwardAPI + - emptyDir + - persistentVolumeClaim + - projected + - secret +groups: [] +# the discovery Jobs run under the namespace default service account +users: + - system:serviceaccount:aim-system:default diff --git a/docs/openshift/extra/local-path-provisioner-scc.yaml b/docs/openshift/extra/02-local-path-provisioner-scc.yaml similarity index 100% rename from docs/openshift/extra/local-path-provisioner-scc.yaml rename to docs/openshift/extra/02-local-path-provisioner-scc.yaml diff --git a/docs/openshift/extra/local-path-helper-pod-selinux.yaml b/docs/openshift/extra/03-local-path-helper-pod-selinux.yaml similarity index 100% rename from docs/openshift/extra/local-path-helper-pod-selinux.yaml rename to docs/openshift/extra/03-local-path-helper-pod-selinux.yaml diff --git a/docs/openshift/extra/local-path-access-mode-scoped.yaml b/docs/openshift/extra/04-local-path-access-mode-scoped.yaml similarity index 100% rename from docs/openshift/extra/local-path-access-mode-scoped.yaml rename to docs/openshift/extra/04-local-path-access-mode-scoped.yaml diff --git a/docs/openshift/extra/05-kyverno.yaml b/docs/openshift/extra/05-kyverno.yaml new file mode 100644 index 00000000..cca84401 --- /dev/null +++ b/docs/openshift/extra/05-kyverno.yaml @@ -0,0 +1,402 @@ +# ============================================================================= +# Extra Kyverno policies for OpenShift +# ============================================================================= +# Everything install.sh applies in STEP 9, in one file because it is one apply: +# a single `ssa_apply` in source order. Document order is therefore load-bearing +# — each RBAC ClusterRole must precede the policy that depends on it, or Kyverno +# rejects the policy at admission. +# +# 1. generate-scc-on-namespaces per-project SCC for labelled Namespaces +# 2. RBAC: routes lets Kyverno manage OpenShift Routes +# 3. generate-routes-from-httproutes HTTPRoute -> Route (non-rewrite) +# 4. ...-rewrite HTTPRoute -> Route (URLRewrite variant) +# 5. RBAC: sccs (cleanup controller) lets the cleanup controller delete SCCs +# 6. cleanup-orphaned-sccs GC for the SCCs generated by (1) +# +# The RBAC for (1) is not here: it is the kyverno-scc-generator ClusterRole in +# extra/scc.yaml, applied back in STEP 2. +# +# NOTE: the two HTTPRoute policies use the placeholder, substituted at +# apply time. No other document in this file contains it. To apply by hand: +# sed 's||apps..example.com|g' 05-kyverno.yaml | kubectl apply --server-side -f - + +# ----------------------------------------------------------------------------- +# 1. generate-scc-on-namespaces +# ----------------------------------------------------------------------------- +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: generate-scc-on-namespaces +spec: + rules: + - name: generate-scc-on-namespaces-rule + match: + any: + - resources: + kinds: + - Namespace + selector: + matchExpressions: + - key: airm.silogen.ai/project-id # Only airm business logics will be applied + operator: Exists + generate: + apiVersion: security.openshift.io/v1 + kind: SecurityContextConstraints + name: "cf-{{request.object.metadata.name}}-scc" + synchronize: true + data: + metadata: + name: "cf-{{request.object.metadata.name}}-scc" + priority: 20 + allowPrivilegedContainer: false + allowPrivilegeEscalation: true + readOnlyRootFilesystem: false + runAsUser: + type: RunAsAny + seLinuxContext: + type: RunAsAny + fsGroup: + type: RunAsAny + supplementalGroups: + type: RunAsAny + seccompProfiles: + - runtime/default + requiredDropCapabilities: [] + allowedCapabilities: + - SETUID + - SETGID + defaultAddCapabilities: + - SETUID + - SETGID + volumes: + - configMap + - downwardAPI + - emptyDir + - ephemeral + - persistentVolumeClaim + - projected + - secret + users: + - "system:serviceaccount:{{request.object.metadata.name}}:default" + +# ----------------------------------------------------------------------------- +# 2. RBAC: let Kyverno manage OpenShift Routes +# ----------------------------------------------------------------------------- +--- +# Grants the Kyverno admission controller list/get on Routes so the +# validate-policy.kyverno.svc webhook accepts ClusterPolicies that +# reference route.openshift.io Routes. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: kyverno:additional:admission-controller:routes + labels: + app.kubernetes.io/component: admission-controller + app.kubernetes.io/instance: kyverno + app.kubernetes.io/part-of: kyverno + rbac.kyverno.io/aggregate-to-admission-controller: "true" +rules: + - apiGroups: ["route.openshift.io"] + resources: ["routes"] + verbs: ["get", "list", "watch"] + +--- +# Grants the Kyverno background controller the permissions it needs to +# actually generate, sync and clean up Routes created by the +# generate-routes-from-httproutes ClusterPolicy. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: kyverno:additional:background-controller:routes + labels: + app.kubernetes.io/component: background-controller + app.kubernetes.io/instance: kyverno + app.kubernetes.io/part-of: kyverno + rbac.kyverno.io/aggregate-to-background-controller: "true" +rules: + - apiGroups: ["route.openshift.io"] + resources: ["routes"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"] + # OpenShift gates `spec.host` on Routes behind a virtual `routes/custom-host` + # subresource. Without this, generated Routes are rejected by the OpenShift + # route admission plugin with: "spec.host: Forbidden: you do not have + # permission to set the host field of the route". + - apiGroups: ["route.openshift.io"] + resources: ["routes/custom-host"] + verbs: ["create", "update"] +# ----------------------------------------------------------------------------- +# 3. generate-routes-from-httproutes (non-rewrite) +# ----------------------------------------------------------------------------- +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: generate-routes-from-httproutes + annotations: + policies.kyverno.io/title: Generate Routes from HTTPRoutes + policies.kyverno.io/category: Generation + policies.kyverno.io/subject: Route + policies.kyverno.io/description: >- + This policy automatically generates OpenShift Routes when Gateway API + HTTPRoutes with airm.silogen.ai/workload-id label are created. +spec: + rules: + - name: generate-route-rule + match: + any: + - resources: + kinds: + - HTTPRoute + selector: + matchExpressions: + - key: airm.silogen.ai/workload-id + operator: Exists + # Skip HTTPRoutes that ask for a path rewrite — those are handled + # by `generate-routes-from-httproutes-rewrite`, which adds the + # haproxy `rewrite-target: /` annotation. Dispatching by the + # `URLRewrite` filter (an intrinsic property of the HTTPRoute) + # instead of by name keeps this generic for any workload type + # (mlflow, vscode, jupyter, ...). + preconditions: + all: + - key: "{{ length(request.object.spec.rules[].filters[?type=='URLRewrite'][]) }}" + operator: Equals + value: 0 + generate: + # Keep generated Routes in sync with their source HTTPRoute, and let + # the background controller self-heal if Kyverno's admission webhook + # was unavailable when the HTTPRoute was created/updated. + synchronize: true + apiVersion: route.openshift.io/v1 + kind: Route + name: "{{request.object.metadata.name}}" + namespace: "{{request.object.metadata.namespace}}" + data: + metadata: + name: "{{request.object.metadata.name}}" + namespace: "{{request.object.metadata.namespace}}" + annotations: + # A Route has no field equivalent to the HTTPRoute's + # spec.rules[].timeouts.request, so without this the generated Route + # silently falls back to the router default of 30s and long-running + # requests die with a 504. AIM sets "30m0s" on its HTTPRoutes, but + # haproxy rejects compound Go durations and drops the annotation + # without logging, so emit the normalised equivalent instead of + # copying the source value. + haproxy.router.openshift.io/timeout: 30m + spec: + # Mirror the source HTTPRoute's own hostname onto the generated Route + # (AIWB sets exactly one, e.g. workspaces.) so the Route serves + # the exact URL the AIWB app advertises. Do NOT hard-code a host here: + # hard-coding aiwbui. produced Routes on the wrong host and made + # the advertised workspace links 503. + host: "{{ request.object.spec.hostnames[0] }}" + path: "{{request.object.spec.rules[0].matches[0].path.value}}" + to: + kind: Service + name: "{{request.object.spec.rules[0].backendRefs[0].name}}" + weight: 100 + port: + targetPort: http + tls: + termination: edge + insecureEdgeTerminationPolicy: Redirect + wildcardPolicy: None +# ----------------------------------------------------------------------------- +# 4. generate-routes-from-httproutes-rewrite +# ----------------------------------------------------------------------------- +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + # Distinct name from the non-rewrite policy so both can co-exist; + # otherwise `kubectl apply -f` of both files silently overwrites one. + name: generate-routes-from-httproutes-rewrite + annotations: + policies.kyverno.io/title: Generate Rewrite Routes from HTTPRoutes + policies.kyverno.io/category: Generation + policies.kyverno.io/subject: Route + policies.kyverno.io/description: >- + Generates an OpenShift Route with the haproxy + `rewrite-target: /` annotation when an AIRM-owned Gateway API + HTTPRoute has a `URLRewrite` filter — i.e. the source HTTPRoute + strips its long path prefix before forwarding to the backend, so + the OpenShift router must do the same. Mutually exclusive with + `generate-routes-from-httproutes` (which handles the no-rewrite + case) via opposite preconditions on the URLRewrite filter. +spec: + rules: + - name: generate-route-rule + match: + any: + - resources: + kinds: + - HTTPRoute + selector: + matchExpressions: + - key: airm.silogen.ai/workload-id + operator: Exists + # Only fire when the source HTTPRoute requests a path rewrite. + # Mutually exclusive with `generate-routes-from-httproutes`. + preconditions: + all: + - key: "{{ length(request.object.spec.rules[].filters[?type=='URLRewrite'][]) }}" + operator: GreaterThanOrEquals + value: 1 + generate: + # Keep generated Routes in sync with their source HTTPRoute, and let + # the background controller self-heal if Kyverno's admission webhook + # was unavailable when the HTTPRoute was created/updated. + synchronize: true + apiVersion: route.openshift.io/v1 + kind: Route + name: "{{request.object.metadata.name}}" + namespace: "{{request.object.metadata.namespace}}" + data: + metadata: + name: "{{request.object.metadata.name}}" + namespace: "{{request.object.metadata.namespace}}" + annotations: + haproxy.router.openshift.io/rewrite-target: / + # A Route has no field equivalent to the HTTPRoute's + # spec.rules[].timeouts.request, so without this the generated Route + # silently falls back to the router default of 30s and long inference + # requests die with a 504. AIM sets "30m0s" on its HTTPRoutes, but + # haproxy rejects compound Go durations and drops the annotation + # without logging, so emit the normalised equivalent instead of + # copying the source value. + haproxy.router.openshift.io/timeout: 30m + spec: + # Mirror the source HTTPRoute's own hostname onto the generated Route + # (AIWB sets exactly one, e.g. workspaces.) so the Route serves + # the exact URL the AIWB app advertises. Do NOT hard-code a host here: + # hard-coding aiwbui. produced Routes on the wrong host and made + # the advertised workspace links 503. + host: "{{ request.object.spec.hostnames[0] }}" + path: "{{request.object.spec.rules[0].matches[0].path.value}}" + to: + kind: Service + name: "{{request.object.spec.rules[0].backendRefs[0].name}}" + weight: 100 + port: + targetPort: http + tls: + termination: edge + insecureEdgeTerminationPolicy: Redirect + wildcardPolicy: None + +# ----------------------------------------------------------------------------- +# 5. RBAC: let the Kyverno cleanup controller delete SCCs +# ----------------------------------------------------------------------------- +--- +# Grants the Kyverno cleanup controller permission to delete SCCs. +# +# The cleanup-controller.kyverno.svc webhook validates a ClusterCleanupPolicy +# at admission time by checking that the cleanup controller's ServiceAccount +# can delete every kind the policy matches. Without this ClusterRole the +# cleanup-orphaned-sccs policy is rejected with: +# "cleanup controller has no permission to delete kind SecurityContextConstraints" +# +# Must be applied BEFORE the cleanup-orphaned-sccs policy, hence its position +# immediately above it in this file. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: kyverno:additional:cleanup-controller:sccs + labels: + app.kubernetes.io/component: cleanup-controller + app.kubernetes.io/instance: kyverno + app.kubernetes.io/part-of: kyverno + rbac.kyverno.io/aggregate-to-cleanup-controller: "true" +rules: + - apiGroups: ["security.openshift.io"] + resources: ["securitycontextconstraints"] + verbs: ["get", "list", "watch", "delete"] + +# ----------------------------------------------------------------------------- +# 6. cleanup-orphaned-sccs +# ----------------------------------------------------------------------------- +--- +# Pre-fetch every Namespace once into Kyverno's global context, list-watch +# style. Without this, the cleanup policy would have to look up the +# source namespace per SCC via `context.apiCall`, but Kyverno caches +# `apiCall` results across all matched resources in a single tick — i.e. +# it fires the apiCall once with the FIRST SCC's namespace name and +# reuses that result for every other SCC. That makes per-resource +# orphan detection impossible (and is actively dangerous: depending on +# iteration order, a stale cached "ns gone" result can cause healthy +# SCCs to be deleted by mistake). A `globalReference` is JMESPath- +# filtered per-resource, so each SCC gets its own correct lookup. +apiVersion: kyverno.io/v2alpha1 +kind: GlobalContextEntry +metadata: + name: namespaces +spec: + kubernetesResource: + group: "" + version: v1 + resource: namespaces +--- +apiVersion: kyverno.io/v2 +kind: ClusterCleanupPolicy +metadata: + name: cleanup-orphaned-sccs + annotations: + policies.kyverno.io/title: Cleanup Orphaned SCCs + policies.kyverno.io/category: Cleanup + policies.kyverno.io/subject: SecurityContextConstraints + policies.kyverno.io/description: >- + Removes orphaned SecurityContextConstraints generated by the + `generate-scc-on-namespaces` policy when their source namespace is + gone or has been recreated with a different UID since the SCC was + generated. +spec: + schedule: "*/1 * * * *" + match: + any: + - resources: + kinds: + - SecurityContextConstraints + selector: + matchLabels: + app.kubernetes.io/managed-by: kyverno + # Must equal the GENERATING ClusterPolicy's name exactly -- Kyverno + # derives this label from it. It read "-activity" here while the + # policy in section 1 of this file is plain + # generate-scc-on-namespaces, so the selector matched nothing and + # the cleanup ran every minute against an empty candidate list. + generate.kyverno.io/policy-name: generate-scc-on-namespaces + context: + - name: namespacesList + globalReference: + name: namespaces + jmesPath: "@" + conditions: + all: + # Single condition that resolves the source namespace's CURRENT UID + # via the globalReference (or 'GONE' if the namespace has been + # deleted) and compares it to the UID the SCC was generated for. + # NotEquals handles both cases: + # - namespace gone -> 'GONE' != trig-uid -> delete + # - namespace recreated -> new-uid != trig-uid -> delete + # - namespace unchanged -> trig-uid == trig-uid -> keep + # + # IMPORTANT — gotchas baked into this single condition: + # + # 1. The matched resource is bound to `target` (NOT `request.object`, + # which is admission-only and silently resolves to empty here — + # that was the original bug that made this policy a no-op). + # + # 2. `split` is a 2-arg function in Kyverno JMESPath: + # `split(s, sep)`. The pipe form `s | split(sep)` silently + # produces nothing (no error logged). + # + # 3. We deliberately do NOT split this into two conditions + # (e.g. a `starts_with` sanity check + the orphan check). Kyverno + # only loads `context.globalReference` for the FIRST condition + # that references it; if a context-free condition is listed + # earlier, the context-using condition gets silently skipped. + - key: '{{ namespacesList[?metadata.name == ''{{ split(target.users[0], '':'')[2] }}''].metadata.uid | [0] || ''GONE'' }}' + operator: NotEquals + value: '{{ target.metadata.labels."generate.kyverno.io/trigger-uid" }}' diff --git a/docs/openshift/extra/06-ai-gateway.yaml b/docs/openshift/extra/06-ai-gateway.yaml new file mode 100644 index 00000000..6cf8dc52 --- /dev/null +++ b/docs/openshift/extra/06-ai-gateway.yaml @@ -0,0 +1,179 @@ +# ============================================================================= +# Envoy AI Gateway — the OpenShift-specific pieces +# ============================================================================= +# Everything else (GatewayClass, GatewayConfig, Gateway, EnvoyProxy, Service, +# EnvoyExtensionPolicy) is rendered from the unmodified cluster-forge chart +# sources/envoy-gateway-config. Only what has no cluster-forge equivalent lives +# here, so the chart stays reusable for RKE2 without a fork. +# +# NOTE: `host` uses the ${DOMAIN} placeholder, substituted at apply time (same +# convention as routes.yaml). To apply by hand: +# DOMAIN=apps..example.com envsubst '${DOMAIN}' < 06-ai-gateway.yaml | kubectl apply -f - +--- +apiVersion: v1 +kind: Namespace +metadata: + name: envoy-gateway-system + +--- +apiVersion: v1 +kind: Namespace +metadata: + name: envoy-ai-gateway-system + +--- +# envoy-gateway-system already gets its anyuid binding from extra/scc.yaml; this +# namespace does not, and it is where the AI controller and the generated +# Endpoint Picker pods run. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: anyuid-scc-envoy-ai-gateway-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: system:openshift:scc:anyuid +subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts:envoy-ai-gateway-system + +--- +# anyuid alone is not enough for these pods, and the failure is easy to +# misread: every Envoy Gateway pod sets seccompProfile RuntimeDefault, and +# anyuid ships with an empty seccompProfiles list, so it rejects them with +# metadata.annotations[container.seccomp...]: Forbidden: seccomp may not be set +# while resolving runAsUser 65532 just fine. The pod is then refused by every +# SCC and the Deployment sits at ReplicaFailure/FailedCreate with no pod and no +# obvious culprit. Same problem extra/scc.yaml already solves for the +# OpenTelemetry operator, solved the same way: a scoped SCC that permits both. +# +# Scoped to the service-account groups rather than named users on purpose — +# Envoy Gateway mints a service account per data-plane Deployment with a +# generated name (envoy---), so there is nothing stable to +# enumerate. Both namespaces are infrastructure we own. +# +# Otherwise as restrictive as restricted-v2: no privilege escalation, no host +# access, all capabilities dropped. +apiVersion: security.openshift.io/v1 +kind: SecurityContextConstraints +metadata: + name: cf-envoy-gateway-scc +priority: 100 +allowHostDirVolumePlugin: false +allowHostIPC: false +allowHostNetwork: false +allowHostPID: false +allowHostPorts: false +allowPrivilegeEscalation: false +allowPrivilegedContainer: false +allowedCapabilities: [] +defaultAddCapabilities: null +requiredDropCapabilities: + - ALL +readOnlyRootFilesystem: false +runAsUser: + type: RunAsAny +seLinuxContext: + type: RunAsAny +seccompProfiles: + - '*' +fsGroup: + type: RunAsAny +supplementalGroups: + type: RunAsAny +groups: + - system:serviceaccounts:envoy-gateway-system + - system:serviceaccounts:envoy-ai-gateway-system +users: [] +volumes: + - configMap + - downwardAPI + - emptyDir + - ephemeral + - persistentVolumeClaim + - projected + - secret + +--- +# OpenShift enables the OwnerReferencesPermissionEnforcement admission plugin; +# upstream Kubernetes and RKE2 do not. It requires that whoever sets +# blockOwnerDeletion: true on an ownerReference can also set finalizers on the +# owner, and ai-gateway-discovery does exactly that when it creates an +# AIGatewayRoute owned by the InferenceService. +# +# Its own ClusterRole grants serving.kserve.io/inferenceservices but not the +# finalizers subresource, and RBAC does not imply subresources from the parent, +# so on OpenShift every AIGatewayRoute is rejected with +# cannot set blockOwnerDeletion if an ownerReference refers to a resource +# you can't set finalizers on +# and no model is ever reachable through the gateway. cluster-forge has no +# reason to carry this rule, since the plugin is not enabled where it runs. +# +# Beware when checking this by hand: `kubectl auth can-i update +# inferenceservices/finalizers` reads the slash as resource/NAME and answers +# yes regardless. Use --subresource=finalizers. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: ai-gateway-discovery-inferenceservice-finalizers +rules: + - apiGroups: ["serving.kserve.io"] + resources: ["inferenceservices/finalizers"] + verbs: ["update"] + +--- +# The service account is created later, by the ai-gateway-discovery step in +# install.sh. Binding ahead of it is fine: the grant takes effect when the +# account appears. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: ai-gateway-discovery-inferenceservice-finalizers +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: ai-gateway-discovery-inferenceservice-finalizers +subjects: + - kind: ServiceAccount + name: ai-gateway-discovery-sa + namespace: ai-gateway-system + +--- +# Stands in for the ai-passthrough listener + TLSRoute that the reference RKE2 +# clusters put on their `https` apps gateway. Here the HAProxy router is that +# front door: it matches SNI ai.${DOMAIN} and forwards the raw TLS stream to the +# ai-gateway Service without decrypting it. +# +# This is the ONLY object that touches cluster ingress, and it claims a hostname +# nothing else serves, so no existing Route is affected. +# +# Passthrough (rather than edge) keeps the manifest set identical to the other +# clusters and the connection encrypted end to end. The cost is that HAProxy +# presents no certificate of its own, so the `cluster-tls` secret in +# envoy-gateway-system must be valid for this hostname — install.sh copies the +# router's own wildcard certificate there to satisfy that. +# +# timeout-tunnel is the annotation that matters: a passthrough Route is TCP mode, +# where plain `timeout` does not govern a long-lived inference stream. Both are +# in HAProxy duration format ("30m"); Go-style "30m0s" is silently ignored. +apiVersion: route.openshift.io/v1 +kind: Route +metadata: + name: ai-gateway + namespace: envoy-gateway-system + annotations: + haproxy.router.openshift.io/timeout: 30m + haproxy.router.openshift.io/timeout-tunnel: 30m +spec: + host: ai.${DOMAIN} + port: + targetPort: https-443 + tls: + termination: passthrough + insecureEdgeTerminationPolicy: Redirect + to: + kind: Service + name: ai-gateway + weight: 100 + wildcardPolicy: None diff --git a/docs/openshift/extra/07-ai-gateway-values.yaml b/docs/openshift/extra/07-ai-gateway-values.yaml new file mode 100644 index 00000000..85b65c44 --- /dev/null +++ b/docs/openshift/extra/07-ai-gateway-values.yaml @@ -0,0 +1,51 @@ +# Helm values for the envoy-gateway chart (sources/envoy-gateway/v1.7.1). +# +# Extracted VERBATIM from cluster-forge root/values.yaml, key: +# apps.envoy-gateway.valuesObject +# +# Regenerate after a cluster-forge bump with: +# yq -y '.apps["envoy-gateway"].valuesObject' /root/values.yaml +# +# The extensionManager block is load-bearing: it wires envoy-gateway to the AI +# controller. Without it AIGatewayRoute is never translated and InferencePool +# routes 503, because nothing populates x-gateway-destination-endpoint. Do not +# hand-trim it, and re-run the command above after every cluster-forge bump: +# nothing checks this file against upstream, and drift here surfaces much later +# as models that resolve but never answer. + +kubernetesClusterDomain: cluster.local +config: + envoyGateway: + extensionApis: + enableBackend: true + extensionManager: + failOpen: true + resources: + - group: aigateway.envoyproxy.io + version: v1beta1 + kind: AIGatewayRoute + - group: aigateway.envoyproxy.io + version: v1beta1 + kind: AIServiceBackend + backendResources: + - group: inference.networking.k8s.io + version: v1 + kind: InferencePool + hooks: + xdsTranslator: + translation: + listener: + includeAll: true + route: + includeAll: true + cluster: + includeAll: true + secret: + includeAll: true + post: + - Translation + - Cluster + - Route + service: + host: ai-gateway-controller.envoy-ai-gateway-system.svc.cluster.local + port: 1063 diff --git a/docs/openshift/extra/amd-gpu-nodefeaturerule.yaml b/docs/openshift/extra/08-amd-gpu-nodefeaturerule.yaml similarity index 100% rename from docs/openshift/extra/amd-gpu-nodefeaturerule.yaml rename to docs/openshift/extra/08-amd-gpu-nodefeaturerule.yaml diff --git a/docs/openshift/extra/routes.yaml b/docs/openshift/extra/09-routes.yaml similarity index 100% rename from docs/openshift/extra/routes.yaml rename to docs/openshift/extra/09-routes.yaml diff --git a/docs/openshift/extra/kyverno-cleanup-policy.yaml b/docs/openshift/extra/kyverno-cleanup-policy.yaml deleted file mode 100644 index df4ad213..00000000 --- a/docs/openshift/extra/kyverno-cleanup-policy.yaml +++ /dev/null @@ -1,83 +0,0 @@ ---- -# Pre-fetch every Namespace once into Kyverno's global context, list-watch -# style. Without this, the cleanup policy would have to look up the -# source namespace per SCC via `context.apiCall`, but Kyverno caches -# `apiCall` results across all matched resources in a single tick — i.e. -# it fires the apiCall once with the FIRST SCC's namespace name and -# reuses that result for every other SCC. That makes per-resource -# orphan detection impossible (and is actively dangerous: depending on -# iteration order, a stale cached "ns gone" result can cause healthy -# SCCs to be deleted by mistake). A `globalReference` is JMESPath- -# filtered per-resource, so each SCC gets its own correct lookup. -apiVersion: kyverno.io/v2alpha1 -kind: GlobalContextEntry -metadata: - name: namespaces -spec: - kubernetesResource: - group: "" - version: v1 - resource: namespaces ---- -apiVersion: kyverno.io/v2 -kind: ClusterCleanupPolicy -metadata: - name: cleanup-orphaned-sccs - annotations: - policies.kyverno.io/title: Cleanup Orphaned SCCs - policies.kyverno.io/category: Cleanup - policies.kyverno.io/subject: SecurityContextConstraints - policies.kyverno.io/description: >- - Removes orphaned SecurityContextConstraints generated by the - `generate-scc-on-namespaces` policy when their source namespace is - gone or has been recreated with a different UID since the SCC was - generated. -spec: - schedule: "*/1 * * * *" - match: - any: - - resources: - kinds: - - SecurityContextConstraints - selector: - matchLabels: - app.kubernetes.io/managed-by: kyverno - # Must equal the GENERATING ClusterPolicy's name exactly -- Kyverno - # derives this label from it. It read "-activity" here while the - # policy in kyverno-scc-for-ns.yaml is plain - # generate-scc-on-namespaces, so the selector matched nothing and - # the cleanup ran every minute against an empty candidate list. - generate.kyverno.io/policy-name: generate-scc-on-namespaces - context: - - name: namespacesList - globalReference: - name: namespaces - jmesPath: "@" - conditions: - all: - # Single condition that resolves the source namespace's CURRENT UID - # via the globalReference (or 'GONE' if the namespace has been - # deleted) and compares it to the UID the SCC was generated for. - # NotEquals handles both cases: - # - namespace gone -> 'GONE' != trig-uid -> delete - # - namespace recreated -> new-uid != trig-uid -> delete - # - namespace unchanged -> trig-uid == trig-uid -> keep - # - # IMPORTANT — gotchas baked into this single condition: - # - # 1. The matched resource is bound to `target` (NOT `request.object`, - # which is admission-only and silently resolves to empty here — - # that was the original bug that made this policy a no-op). - # - # 2. `split` is a 2-arg function in Kyverno JMESPath: - # `split(s, sep)`. The pipe form `s | split(sep)` silently - # produces nothing (no error logged). - # - # 3. We deliberately do NOT split this into two conditions - # (e.g. a `starts_with` sanity check + the orphan check). Kyverno - # only loads `context.globalReference` for the FIRST condition - # that references it; if a context-free condition is listed - # earlier, the context-using condition gets silently skipped. - - key: '{{ namespacesList[?metadata.name == ''{{ split(target.users[0], '':'')[2] }}''].metadata.uid | [0] || ''GONE'' }}' - operator: NotEquals - value: '{{ target.metadata.labels."generate.kyverno.io/trigger-uid" }}' diff --git a/docs/openshift/extra/kyverno-cleanup-scc-permissions.yaml b/docs/openshift/extra/kyverno-cleanup-scc-permissions.yaml deleted file mode 100644 index 955e7f05..00000000 --- a/docs/openshift/extra/kyverno-cleanup-scc-permissions.yaml +++ /dev/null @@ -1,23 +0,0 @@ ---- -# Grants the Kyverno cleanup controller permission to delete SCCs. -# -# The cleanup-controller.kyverno.svc webhook validates a ClusterCleanupPolicy -# at admission time by checking that the cleanup controller's ServiceAccount -# can delete every kind the policy matches. Without this ClusterRole the -# cleanup-orphaned-sccs policy is rejected with: -# "cleanup controller has no permission to delete kind SecurityContextConstraints" -# -# Must be applied BEFORE kyverno-cleanup-policy.yaml. -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: kyverno:additional:cleanup-controller:sccs - labels: - app.kubernetes.io/component: cleanup-controller - app.kubernetes.io/instance: kyverno - app.kubernetes.io/part-of: kyverno - rbac.kyverno.io/aggregate-to-cleanup-controller: "true" -rules: - - apiGroups: ["security.openshift.io"] - resources: ["securitycontextconstraints"] - verbs: ["get", "list", "watch", "delete"] diff --git a/docs/openshift/extra/kyverno-httproute-to-route-non-rewrite-policy.yaml b/docs/openshift/extra/kyverno-httproute-to-route-non-rewrite-policy.yaml deleted file mode 100644 index be99ed3f..00000000 --- a/docs/openshift/extra/kyverno-httproute-to-route-non-rewrite-policy.yaml +++ /dev/null @@ -1,75 +0,0 @@ ---- -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: generate-routes-from-httproutes - annotations: - policies.kyverno.io/title: Generate Routes from HTTPRoutes - policies.kyverno.io/category: Generation - policies.kyverno.io/subject: Route - policies.kyverno.io/description: >- - This policy automatically generates OpenShift Routes when Gateway API - HTTPRoutes with airm.silogen.ai/workload-id label are created. -spec: - rules: - - name: generate-route-rule - match: - any: - - resources: - kinds: - - HTTPRoute - selector: - matchExpressions: - - key: airm.silogen.ai/workload-id - operator: Exists - # Skip HTTPRoutes that ask for a path rewrite — those are handled - # by `generate-routes-from-httproutes-rewrite`, which adds the - # haproxy `rewrite-target: /` annotation. Dispatching by the - # `URLRewrite` filter (an intrinsic property of the HTTPRoute) - # instead of by name keeps this generic for any workload type - # (mlflow, vscode, jupyter, ...). - preconditions: - all: - - key: "{{ length(request.object.spec.rules[].filters[?type=='URLRewrite'][]) }}" - operator: Equals - value: 0 - generate: - # Keep generated Routes in sync with their source HTTPRoute, and let - # the background controller self-heal if Kyverno's admission webhook - # was unavailable when the HTTPRoute was created/updated. - synchronize: true - apiVersion: route.openshift.io/v1 - kind: Route - name: "{{request.object.metadata.name}}" - namespace: "{{request.object.metadata.namespace}}" - data: - metadata: - name: "{{request.object.metadata.name}}" - namespace: "{{request.object.metadata.namespace}}" - annotations: - # A Route has no field equivalent to the HTTPRoute's - # spec.rules[].timeouts.request, so without this the generated Route - # silently falls back to the router default of 30s and long-running - # requests die with a 504. AIM sets "30m0s" on its HTTPRoutes, but - # haproxy rejects compound Go durations and drops the annotation - # without logging, so emit the normalised equivalent instead of - # copying the source value. - haproxy.router.openshift.io/timeout: 30m - spec: - # Mirror the source HTTPRoute's own hostname onto the generated Route - # (AIWB sets exactly one, e.g. workspaces.) so the Route serves - # the exact URL the AIWB app advertises. Do NOT hard-code a host here: - # hard-coding aiwbui. produced Routes on the wrong host and made - # the advertised workspace links 503. - host: "{{ request.object.spec.hostnames[0] }}" - path: "{{request.object.spec.rules[0].matches[0].path.value}}" - to: - kind: Service - name: "{{request.object.spec.rules[0].backendRefs[0].name}}" - weight: 100 - port: - targetPort: http - tls: - termination: edge - insecureEdgeTerminationPolicy: Redirect - wildcardPolicy: None \ No newline at end of file diff --git a/docs/openshift/extra/kyverno-httproute-to-route-rewrite-policy.yaml b/docs/openshift/extra/kyverno-httproute-to-route-rewrite-policy.yaml deleted file mode 100644 index 04dc3b0e..00000000 --- a/docs/openshift/extra/kyverno-httproute-to-route-rewrite-policy.yaml +++ /dev/null @@ -1,79 +0,0 @@ ---- -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - # Distinct name from the non-rewrite policy so both can co-exist; - # otherwise `kubectl apply -f` of both files silently overwrites one. - name: generate-routes-from-httproutes-rewrite - annotations: - policies.kyverno.io/title: Generate Rewrite Routes from HTTPRoutes - policies.kyverno.io/category: Generation - policies.kyverno.io/subject: Route - policies.kyverno.io/description: >- - Generates an OpenShift Route with the haproxy - `rewrite-target: /` annotation when an AIRM-owned Gateway API - HTTPRoute has a `URLRewrite` filter — i.e. the source HTTPRoute - strips its long path prefix before forwarding to the backend, so - the OpenShift router must do the same. Mutually exclusive with - `generate-routes-from-httproutes` (which handles the no-rewrite - case) via opposite preconditions on the URLRewrite filter. -spec: - rules: - - name: generate-route-rule - match: - any: - - resources: - kinds: - - HTTPRoute - selector: - matchExpressions: - - key: airm.silogen.ai/workload-id - operator: Exists - # Only fire when the source HTTPRoute requests a path rewrite. - # Mutually exclusive with `generate-routes-from-httproutes`. - preconditions: - all: - - key: "{{ length(request.object.spec.rules[].filters[?type=='URLRewrite'][]) }}" - operator: GreaterThanOrEquals - value: 1 - generate: - # Keep generated Routes in sync with their source HTTPRoute, and let - # the background controller self-heal if Kyverno's admission webhook - # was unavailable when the HTTPRoute was created/updated. - synchronize: true - apiVersion: route.openshift.io/v1 - kind: Route - name: "{{request.object.metadata.name}}" - namespace: "{{request.object.metadata.namespace}}" - data: - metadata: - name: "{{request.object.metadata.name}}" - namespace: "{{request.object.metadata.namespace}}" - annotations: - haproxy.router.openshift.io/rewrite-target: / - # A Route has no field equivalent to the HTTPRoute's - # spec.rules[].timeouts.request, so without this the generated Route - # silently falls back to the router default of 30s and long inference - # requests die with a 504. AIM sets "30m0s" on its HTTPRoutes, but - # haproxy rejects compound Go durations and drops the annotation - # without logging, so emit the normalised equivalent instead of - # copying the source value. - haproxy.router.openshift.io/timeout: 30m - spec: - # Mirror the source HTTPRoute's own hostname onto the generated Route - # (AIWB sets exactly one, e.g. workspaces.) so the Route serves - # the exact URL the AIWB app advertises. Do NOT hard-code a host here: - # hard-coding aiwbui. produced Routes on the wrong host and made - # the advertised workspace links 503. - host: "{{ request.object.spec.hostnames[0] }}" - path: "{{request.object.spec.rules[0].matches[0].path.value}}" - to: - kind: Service - name: "{{request.object.spec.rules[0].backendRefs[0].name}}" - weight: 100 - port: - targetPort: http - tls: - termination: edge - insecureEdgeTerminationPolicy: Redirect - wildcardPolicy: None diff --git a/docs/openshift/extra/kyverno-route-permissions.yaml b/docs/openshift/extra/kyverno-route-permissions.yaml deleted file mode 100644 index 65f5a0f4..00000000 --- a/docs/openshift/extra/kyverno-route-permissions.yaml +++ /dev/null @@ -1,42 +0,0 @@ ---- -# Grants the Kyverno admission controller list/get on Routes so the -# validate-policy.kyverno.svc webhook accepts ClusterPolicies that -# reference route.openshift.io Routes. -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: kyverno:additional:admission-controller:routes - labels: - app.kubernetes.io/component: admission-controller - app.kubernetes.io/instance: kyverno - app.kubernetes.io/part-of: kyverno - rbac.kyverno.io/aggregate-to-admission-controller: "true" -rules: - - apiGroups: ["route.openshift.io"] - resources: ["routes"] - verbs: ["get", "list", "watch"] - ---- -# Grants the Kyverno background controller the permissions it needs to -# actually generate, sync and clean up Routes created by the -# generate-routes-from-httproutes ClusterPolicy. -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: kyverno:additional:background-controller:routes - labels: - app.kubernetes.io/component: background-controller - app.kubernetes.io/instance: kyverno - app.kubernetes.io/part-of: kyverno - rbac.kyverno.io/aggregate-to-background-controller: "true" -rules: - - apiGroups: ["route.openshift.io"] - resources: ["routes"] - verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"] - # OpenShift gates `spec.host` on Routes behind a virtual `routes/custom-host` - # subresource. Without this, generated Routes are rejected by the OpenShift - # route admission plugin with: "spec.host: Forbidden: you do not have - # permission to set the host field of the route". - - apiGroups: ["route.openshift.io"] - resources: ["routes/custom-host"] - verbs: ["create", "update"] \ No newline at end of file diff --git a/docs/openshift/extra/kyverno-scc-for-ns.yaml b/docs/openshift/extra/kyverno-scc-for-ns.yaml deleted file mode 100644 index f954a9f8..00000000 --- a/docs/openshift/extra/kyverno-scc-for-ns.yaml +++ /dev/null @@ -1,56 +0,0 @@ ---- -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: generate-scc-on-namespaces -spec: - rules: - - name: generate-scc-on-namespaces-rule - match: - any: - - resources: - kinds: - - Namespace - selector: - matchExpressions: - - key: airm.silogen.ai/project-id # Only airm business logics will be applied - operator: Exists - generate: - apiVersion: security.openshift.io/v1 - kind: SecurityContextConstraints - name: "cf-{{request.object.metadata.name}}-scc" - synchronize: true - data: - metadata: - name: "cf-{{request.object.metadata.name}}-scc" - priority: 20 - allowPrivilegedContainer: false - allowPrivilegeEscalation: true - readOnlyRootFilesystem: false - runAsUser: - type: RunAsAny - seLinuxContext: - type: RunAsAny - fsGroup: - type: RunAsAny - supplementalGroups: - type: RunAsAny - seccompProfiles: - - runtime/default - requiredDropCapabilities: [] - allowedCapabilities: - - SETUID - - SETGID - defaultAddCapabilities: - - SETUID - - SETGID - volumes: - - configMap - - downwardAPI - - emptyDir - - ephemeral - - persistentVolumeClaim - - projected - - secret - users: - - "system:serviceaccount:{{request.object.metadata.name}}:default" diff --git a/docs/openshift/extra/scc-aim-system.yaml b/docs/openshift/extra/scc-aim-system.yaml deleted file mode 100644 index e9c9c02d..00000000 --- a/docs/openshift/extra/scc-aim-system.yaml +++ /dev/null @@ -1,116 +0,0 @@ -# Extra SecurityContextConstraints for the aim-system namespace. -# -# scc.yaml already grants anyuid to system:serviceaccounts:aim-system -# (ClusterRoleBinding anyuid-scc-aim-system). anyuid allows an arbitrary UID but -# NOT hostPath volumes, NOT privileged containers and NOT seccomp profiles, so -# the aim-engine accelerator-detector DaemonSets and the model discovery Jobs -# are rejected at admission and never create a pod at all: -# -# daemonset/...-accelerator-detector-cpu DESIRED 1 CURRENT 0 (466 x FailedCreate) -# daemonset/...-accelerator-detector-gpu DESIRED 1 CURRENT 0 (464 x FailedCreate) -# Error creating: pods "..." is forbidden: unable to validate against any -# security context constraint: ... hostPath volumes are not allowed to be used -# -# Both SCCs below are bound to named service accounts via users:, never to a -# group, so no other workload in the cluster can select them. -# -# priority is deliberately left unset. SCC selection then prefers the most -# restrictive constraint that can admit the pod, so these are only reached after -# restricted-v2 and anyuid have already failed to validate. ---- -apiVersion: security.openshift.io/v1 -kind: SecurityContextConstraints -metadata: - name: cf-aim-engine-detector-scc - annotations: - kubernetes.io/description: >- - Lets the aim-engine accelerator-detector DaemonSets mount host paths and, - for the GPU variant, run privileged. Bound to one service account. -# Required by both DaemonSets: -# /etc/kubernetes/node-feature-discovery/features.d/ writes NFD feature files -# /proc/cpuinfo cpu variant -# /dev/kfd, /dev/dri gpu variant -allowHostDirVolumePlugin: true -allowHostIPC: false -allowHostNetwork: false -allowHostPID: false -allowHostPorts: false -# gpu variant sets containers[0].securityContext.privileged: true so it can -# reach /dev/kfd and /dev/dri -allowPrivilegedContainer: true -allowPrivilegeEscalation: true -# Neither pod adds a capability: the cpu detector drops ALL itself and the gpu -# detector relies on privileged, so nothing needs to be granted or forced. -allowedCapabilities: [] -defaultAddCapabilities: [] -requiredDropCapabilities: [] -readOnlyRootFilesystem: false -# initContainer prep-features-dir runs as 0, detector container as 65534 -runAsUser: - type: RunAsAny -seLinuxContext: - type: RunAsAny -fsGroup: - type: RunAsAny -supplementalGroups: - type: RunAsAny -seccompProfiles: - - '*' -userNamespaceLevel: AllowHostLevel -volumes: - - configMap - - downwardAPI - - emptyDir - - hostPath - - persistentVolumeClaim - - projected - - secret -groups: [] -users: - - system:serviceaccount:aim-system:aim-engine-aim-engine-chart-accelerator-detector ---- -apiVersion: security.openshift.io/v1 -kind: SecurityContextConstraints -metadata: - name: cf-aim-engine-discovery-scc - annotations: - kubernetes.io/description: >- - Lets the aim-engine model discovery Jobs set a seccomp profile and run as - UID 65532. No host access of any kind. -allowHostDirVolumePlugin: false -allowHostIPC: false -allowHostNetwork: false -allowHostPID: false -allowHostPorts: false -allowPrivilegedContainer: false -allowPrivilegeEscalation: false -allowedCapabilities: [] -defaultAddCapabilities: [] -requiredDropCapabilities: - - ALL -readOnlyRootFilesystem: false -# pods set runAsUser 65532, outside the namespace-allocated UID range -runAsUser: - type: RunAsAny -seLinuxContext: - type: RunAsAny -fsGroup: - type: RunAsAny -supplementalGroups: - type: RunAsAny -# pods set securityContext.seccompProfile.type: RuntimeDefault, which anyuid -# rejects with "metadata.annotations[seccomp...]: Forbidden: seccomp may not be set" -seccompProfiles: - - runtime/default -userNamespaceLevel: AllowHostLevel -volumes: - - configMap - - downwardAPI - - emptyDir - - persistentVolumeClaim - - projected - - secret -groups: [] -# the discovery Jobs run under the namespace default service account -users: - - system:serviceaccount:aim-system:default diff --git a/docs/openshift/install.sh b/docs/openshift/install.sh index 1af7bd21..bba4e29e 100755 --- a/docs/openshift/install.sh +++ b/docs/openshift/install.sh @@ -65,7 +65,7 @@ EXTRA_RAW_BASE="https://raw.githubusercontent.com/silogen/cluster-forge/refs/hea # AIWB_RELEASE_VERSION_TAG=2.0.0 # NOTE: anyuid SCC grants are no longer applied per-namespace here. They are -# consolidated as declarative ClusterRoleBinding manifests in extra/scc.yaml, +# consolidated as declarative ClusterRoleBinding manifests in extra/01-scc.yaml, # applied up front (see "Applying custom SecurityContextConstraints" below). # ============================================================================ @@ -176,13 +176,84 @@ detect_amd_gpu_ns() { printf '%s' "${ns:-amd-gpu-operator}" } +# cf_app_version : print apps..repoVersion from the downloaded +# release's root/values.yaml. +# +# The charts installed from OCI registries are NOT in the release tarball, and +# their versions do not track the cluster-forge release number at all: at v2.2.2 +# aiwb is 2.0.0, aim-engine is 0.2.5 and kaiwo is v0.2.1. Deriving them from +# CLUSTER_FORGE_VERSION therefore cannot work -- there is no 2.2.2 tag for any of +# them. root/values.yaml is where cluster-forge states which chart version +# belongs to the release, so that is the thing to read. +# +# Hardcoding them here instead means every release bump silently keeps installing +# the previous charts, which looks exactly like the installer having no effect. +# +# Parsed with awk rather than yq: this is the only YAML this script has to read, +# the block is a fixed two-level shape, and a `curl | bash` run cannot assume yq +# is installed. Missing values are fatal, for the same reason as in +# ensure_extra_file -- stopping here is far cheaper than a cluster that came up +# on charts nobody chose. +# +# A per-app override wins over the release, for the case where a chart has been +# published that no cluster-forge release references yet: +# +# CF_VERSION_AIWB=2.0.1 ./install.sh +# +# The variable name is the app key uppercased with dashes turned into +# underscores, so ai-gateway-discovery is CF_VERSION_AI_GATEWAY_DISCOVERY. +# Overriding is a deliberate step outside what the release was tested with, so +# it is announced in the output rather than applied quietly. +cf_app_version() { + local app="$1" v override + override="CF_VERSION_$(printf '%s' "${app}" | tr 'a-z-' 'A-Z_')" + if [ -n "${!override:-}" ]; then + echo "ℹ️ ${app}: using ${!override} from ${override}, overriding cluster-forge ${CLUSTER_FORGE_VERSION}" >&2 + printf '%s' "${!override}" + return 0 + fi + [ -f "${CF_ROOT_VALUES}" ] || { echo "❌ ${CF_ROOT_VALUES} not found; cannot resolve chart versions" >&2; exit 1; } + v=$(awk -v want=" ${app}:" ' + /^apps:/ { inapps = 1; next } + inapps && /^[^ ]/ { exit } # left the apps: block + inapps && $0 == want { inapp = 1; next } + inapp && /^ [^ ]/ { exit } # next app at the same indent + inapp && $1 == "repoVersion:" { gsub(/"/, "", $2); print $2; exit } + ' "${CF_ROOT_VALUES}") + if [ -z "${v}" ]; then + echo "❌ apps.${app}.repoVersion not found in ${CF_ROOT_VALUES}" >&2 + echo " cluster-forge ${CLUSTER_FORGE_VERSION} may have renamed or dropped this app." >&2 + exit 1 + fi + printf '%s' "${v}" +} + +# is_openshift : true when this cluster serves the OpenShift Route API. +# +# Route is an aggregated API served by the openshift-apiserver, NOT a +# CustomResourceDefinition, so the `kubectl get crd ...` test used elsewhere in +# this script finds nothing even on a real OpenShift cluster. Probe the API +# group instead. +# +# Route is also the right thing to probe rather than, say, SecurityContextConstraints: +# the OpenShift-only work in this script is about exposing services through the +# HAProxy router, and a cluster without Routes cannot do any of it. +is_openshift() { + kubectl get --raw /apis/route.openshift.io/v1 >/dev/null 2>&1 +} + # ensure_extra_file : guarantee the manifest is on disk, # fetching it from ${EXTRA_RAW_BASE} when it is not. # +# The filename is also the remote name: extra/ in this repo and extra/ on +# cluster-forge main hold the same files under the same names, NN- prefix +# included. Keep it that way. If the two ever diverge, a piped run stops being +# equivalent to a checkout run, which is the whole point of this function. +# # EVERY extra/ manifest must go through this. A bare `[ -f ... ]` test that # warns and continues is not an acceptable substitute: it makes a piped run # (curl | bash, nothing on disk) skip the manifest entirely, and some of them -# are load-bearing -- without local-path-provisioner-scc.yaml every PVC in the +# are load-bearing -- without 02-local-path-provisioner-scc.yaml every PVC in the # cluster fails at admission, twenty minutes after the warning scrolled past. # # Hence: fetch failure is fatal. Stopping with the filename and URL is far @@ -284,6 +355,31 @@ else AIWB_UI_URL="https://aiwbui.${DOMAIN}" fi +# --- Envoy AI Gateway (see the ENVOY AI GATEWAY step) ------------------------ +# The single hostname every served model is reached on, with the model selected +# per-request by the x-ai-eg-backend / x-ai-eg-model headers. +AI_HOST="ai.${DOMAIN}" + +# The gateway is fronted by an OpenShift Route and needs an OpenShift SCC, so +# the whole step only applies where those exist. Probed once, here, so the +# answer is the same for the source patches and for the step that uses them. +if is_openshift; then + IS_OPENSHIFT=true +else + IS_OPENSHIFT=false + echo "ℹ️ route.openshift.io is not served — the OpenShift-only steps will be skipped" +fi + +# Envoy worker threads. Left unset, Envoy starts one worker per cpuset thread, +# and each worker costs about four file descriptors before serving anything. +# CRI-O hands containers a soft nofile limit of 1024, so any node with >=256 +# cores exhausts it during startup and the data plane never comes up. +ENVOY_CONCURRENCY="${ENVOY_CONCURRENCY:-4}" + +# The AI controller encrypts MCP session state with this seed. cluster-forge +# ships a placeholder; override it on any cluster that will serve real traffic. +AI_GATEWAY_MCP_SEED="${AI_GATEWAY_MCP_SEED:-cluster-forge-default-seed-override-in-production}" + # Download a pinned cluster-forge release tarball instead of cloning a branch. # CLUSTER_FORGE_VERSION selects the GitHub release (e.g. v2.2.0). The release # asset "release-enterprise-ai-.tar.gz" unpacks into a top-level @@ -309,6 +405,8 @@ rm -f "${CLUSTER_FORGE_DIR}/${RELEASE_TARBALL}" # The tarball unpacks under ${CLUSTER_FORGE_DIR}/cluster-forge; sources live there. SOURCES_DIR="${CLUSTER_FORGE_DIR}/cluster-forge/sources" +# Chart versions for the OCI-hosted apps are declared here, not in sources/. +CF_ROOT_VALUES="${CLUSTER_FORGE_DIR}/cluster-forge/root/values.yaml" # EXTRA_DIR is defined near the top, alongside the preflight that verifies the # manifests which have no upstream fallback. @@ -338,6 +436,43 @@ if ! grep -q "failOpen" "${EXTAUTH_TPL}" 2>/dev/null; then echo "✅ Patched envoy-gateway-config SecurityPolicy: failOpen=true" fi +# --- envoy-gateway-config, for the AI gateway on OpenShift ------------------- +# The chart is written for RKE2: a LoadBalancer apps gateway owning the whole +# domain, and ordinary node sizes. Three lines have to change here. Patching the +# extracted sources rather than forking keeps the chart reusable for RKE2; the +# sources are re-downloaded on every run, so these are one-shot edits. +# +# Each patch is asserted afterwards because upstream could rename or reformat +# the line it targets, and every failure mode is slow and misleading: a gateway +# claiming every hostname, a pod Pending forever, a CrashLoopBackOff that reads +# like a networking fault. +if [ "${IS_OPENSHIFT}" = "true" ]; then + AI_GW_TPL="${SOURCES_DIR}/envoy-gateway-config/templates/ai-gateway.yaml" + AI_GW_PROXY_TPL="${SOURCES_DIR}/envoy-gateway-config/templates/ai-gateway-proxy-config.yaml" + + # 1. The listener is "*." because on RKE2 the apps gateway owns the + # whole domain. Here HAProxy owns it and hands over one name, so the + # wildcard would only widen which HTTPRoutes can attach to this gateway. + sed -i 's|hostname: "\*\.{{ \.Values\.domain }}"|hostname: "{{ .Values.aiGateway.routeHostname }}"|' "${AI_GW_TPL}" + grep -q 'hostname: "{{ .Values.aiGateway.routeHostname }}"' "${AI_GW_TPL}" \ + || { echo "❌ ai-gateway.yaml hostname patch did not apply — upstream changed" >&2; exit 1; } + + # 2. Drop the cluster-bloom/first-node nodeSelector. No OpenShift node carries + # that label, so the Envoy pod would sit Pending forever. + sed -i '/^ *nodeSelector:$/{N;/cluster-bloom\/first-node/d}' "${AI_GW_PROXY_TPL}" + ! grep -q 'cluster-bloom/first-node' "${AI_GW_PROXY_TPL}" \ + || { echo "❌ ai-gateway-proxy-config.yaml nodeSelector patch did not apply — upstream changed" >&2; exit 1; } + + # 3. Cap the Envoy worker threads (see ENVOY_CONCURRENCY above). Raising the + # ulimit instead would mean a node-level MachineConfig for CRI-O, a far + # bigger blast radius than bounding a thread count the gateway never needs. + sed -i "0,/^spec:$/s//spec:\n concurrency: ${ENVOY_CONCURRENCY}/" "${AI_GW_PROXY_TPL}" + grep -q "concurrency: ${ENVOY_CONCURRENCY}" "${AI_GW_PROXY_TPL}" \ + || { echo "❌ ai-gateway-proxy-config.yaml concurrency patch did not apply — upstream changed" >&2; exit 1; } + + echo "✅ Patched envoy-gateway-config for OpenShift: exact hostname, no nodeSelector, concurrency=${ENVOY_CONCURRENCY}" +fi + # ============================================================================ # CUSTOM SECURITY CONTEXT CONSTRAINTS (OpenShift) # ============================================================================ @@ -349,19 +484,17 @@ step "Custom SecurityContextConstraints (SCCs)" # they exist before any workload pod is scheduled. SCC `users` entries may point # at service accounts that do not exist yet — that is fine; the binding takes # effect once the SA is created. -SCC_FILE="${EXTRA_DIR}/scc.yaml" +# +# The file's last section covers aim-system: the aim-engine accelerator-detector +# DaemonSets and discovery Jobs mount hostPath (they write NFD feature files +# under /etc/kubernetes/node-feature-discovery/features.d/), and the GPU +# detector also needs privileged to reach /dev/kfd and /dev/dri, so +# restricted-v2 rejects them outright and the DaemonSets sit at DESIRED>0 / +# CURRENT=0. +SCC_FILE="${EXTRA_DIR}/01-scc.yaml" echo "📦 Applying custom SecurityContextConstraints..." ensure_extra_file "${SCC_FILE}" retry kubectl apply --request-timeout="${KUBECTL_REQUEST_TIMEOUT}" -f "${SCC_FILE}" - -# SCCs for the aim-engine accelerator-detector DaemonSets and discovery Jobs. -# Both mount hostPath (they write NFD feature files under -# /etc/kubernetes/node-feature-discovery/features.d/), and the GPU detector also -# needs privileged to reach /dev/kfd and /dev/dri, so restricted-v2 rejects them -# outright and the DaemonSets sit at DESIRED>0 / CURRENT=0. -SCC_AIM_FILE="${EXTRA_DIR}/scc-aim-system.yaml" -ensure_extra_file "${SCC_AIM_FILE}" -retry kubectl apply --request-timeout="${KUBECTL_REQUEST_TIMEOUT}" -f "${SCC_AIM_FILE}" echo "✅ Custom SCCs applied" # ============================================================================ @@ -392,8 +525,8 @@ fi # # Deliberately OUTSIDE the guard above: re-applying the upstream manifest # reverts the ConfigMap patch, so both parts are re-asserted on every run. -LOCAL_PATH_SCC_FILE="${EXTRA_DIR}/local-path-provisioner-scc.yaml" -LOCAL_PATH_HELPER_PATCH_FILE="${EXTRA_DIR}/local-path-helper-pod-selinux.yaml" +LOCAL_PATH_SCC_FILE="${EXTRA_DIR}/02-local-path-provisioner-scc.yaml" +LOCAL_PATH_HELPER_PATCH_FILE="${EXTRA_DIR}/03-local-path-helper-pod-selinux.yaml" if ! kubectl get configmap local-path-config -n local-path-storage &>/dev/null; then # e.g. RKE2, where the built-in provisioner lives in kube-system instead. # Nothing to patch, so the manifests are not fetched either. @@ -604,7 +737,7 @@ helm template kyverno-policies-storage ${SOURCES_DIR}/kyverno-policies/storage-l # the conversion visible (the chart's companion "warning" policy cannot fire -- # see the file header). Must run AFTER the helm template above, which is why it # sits here rather than with the other extra/ policies further down. -ACCESS_MODE_POLICY_FILE="${EXTRA_DIR}/local-path-access-mode-scoped.yaml" +ACCESS_MODE_POLICY_FILE="${EXTRA_DIR}/04-local-path-access-mode-scoped.yaml" if ! kubectl get clusterpolicy local-path-access-mode-mutation &>/dev/null; then # Chart not deployed on this cluster size (it ships only to small/medium), # so there is nothing to scope. @@ -622,63 +755,48 @@ echo "" # EXTRA OPENSHIFT KYVERNO POLICIES # ============================================================================ step "Extra OpenShift Kyverno policies (SCC + HTTPRoute->Route)" -# generate-scc-on-namespaces: auto-generates a per-project OpenShift SCC for any -# Namespace labelled airm.silogen.ai/project-id. The RBAC that lets Kyverno -# manage SecurityContextConstraints (the kyverno-scc-generator ClusterRole and -# binding) is applied earlier from extra/scc.yaml, so by now it already exists. -echo "📦 Installing extra OpenShift Kyverno policies..." -# EXTRA_DIR is defined once near the top (defaults to the extra/ dir beside this script). -KYVERNO_SCC_POLICY_FILE="${EXTRA_DIR}/kyverno-scc-for-ns.yaml" -ensure_extra_file "${KYVERNO_SCC_POLICY_FILE}" -ssa_apply < "${KYVERNO_SCC_POLICY_FILE}" - -# HTTPRoute -> OpenShift Route automation. AIWB exposes workspace apps via -# Gateway API HTTPRoutes (labelled airm.silogen.ai/workload-id). On OpenShift -# there is no Gateway controller serving them, so these Kyverno policies watch -# those HTTPRoutes and generate a matching OpenShift Route automatically. -# 1. RBAC first — lets the Kyverno admission/background controllers manage -# Routes (incl. routes/custom-host, required to set spec.host). -# 2. Policies — the non-rewrite + rewrite variants. They hard-code the host -# as "workloads.", so substitute the real cluster domain on the fly -# (local apply-time patch; sources/extra files stay branch-default). -echo "📦 Installing HTTPRoute->Route Kyverno policies..." +# One file, one apply, in source order. What it contains and why: +# +# generate-scc-on-namespaces — auto-generates a per-project OpenShift SCC for +# any Namespace labelled airm.silogen.ai/project-id. The RBAC that lets +# Kyverno manage SecurityContextConstraints (the kyverno-scc-generator +# ClusterRole and binding) came earlier from extra/01-scc.yaml, so it exists. +# +# HTTPRoute -> OpenShift Route automation — AIWB exposes workspace apps via +# Gateway API HTTPRoutes (labelled airm.silogen.ai/workload-id). On OpenShift +# no Gateway controller serves them, so these policies watch those HTTPRoutes +# and generate a matching Route. RBAC comes first in the file so the +# admission/background controllers can manage Routes (incl. +# routes/custom-host, required to set spec.host) before the generate policies +# are admitted. Both variants (non-rewrite + rewrite) hard-code the host as +# "workloads.", hence the substitution below (apply-time patch only; +# the extra/ file stays branch-default). +# +# cleanup-orphaned-sccs — companion GC for the SCCs generated above. A +# ClusterCleanupPolicy (+ GlobalContextEntry) that runs every minute and +# deletes per-project SCCs whose source namespace is gone or was recreated +# with a different UID, so they do not accumulate over the cluster's +# lifetime. Requires the Kyverno cleanup controller to be enabled. Its RBAC +# also precedes it in the file: the cleanup-controller webhook validates at +# admission that the controller can delete every kind the policy matches, +# otherwise the policy is rejected with "cleanup controller has no permission +# to delete kind SecurityContextConstraints". +# # OpenShift's default route admission is "Strict": a Route in one namespace # cannot claim a hostname already owned by a Route in another namespace. The -# policies below generate workspace Routes on the AIWB UI host (aiwbui., -# owned by the aiwb namespace) under distinct /workbench/<...> paths, but those -# Routes live in the per-workspace namespaces (e.g. workbench). Without this they -# are rejected with "HostAlreadyClaimed". InterNamespaceAllowed permits sharing a +# policies generate workspace Routes on the AIWB UI host (aiwbui., owned +# by the aiwb namespace) under distinct /workbench/<...> paths, but those Routes +# live in the per-workspace namespaces (e.g. workbench). Without this they are +# rejected with "HostAlreadyClaimed". InterNamespaceAllowed permits sharing a # host across namespaces (path-based), which is exactly what AIWB needs. echo "🔧 Allowing inter-namespace route host sharing (workspace Routes share the AIWB UI host)..." retry kubectl patch ingresscontroller default -n openshift-ingress-operator --type=merge \ -p '{"spec":{"routeAdmission":{"namespaceOwnership":"InterNamespaceAllowed"}}}' -KYVERNO_ROUTE_RBAC_FILE="${EXTRA_DIR}/kyverno-route-permissions.yaml" -KYVERNO_ROUTE_POLICY_FILE="${EXTRA_DIR}/kyverno-httproute-to-route-non-rewrite-policy.yaml" -KYVERNO_ROUTE_REWRITE_POLICY_FILE="${EXTRA_DIR}/kyverno-httproute-to-route-rewrite-policy.yaml" -ensure_extra_file "${KYVERNO_ROUTE_RBAC_FILE}" -ensure_extra_file "${KYVERNO_ROUTE_POLICY_FILE}" -ensure_extra_file "${KYVERNO_ROUTE_REWRITE_POLICY_FILE}" -# RBAC must exist before the generate policies so Kyverno can create Routes. -ssa_apply < "${KYVERNO_ROUTE_RBAC_FILE}" -sed "s||${DOMAIN}|g" "${KYVERNO_ROUTE_POLICY_FILE}" | ssa_apply -sed "s||${DOMAIN}|g" "${KYVERNO_ROUTE_REWRITE_POLICY_FILE}" | ssa_apply - -# Orphaned-SCC garbage collector: companion to kyverno-scc-for-ns.yaml. A -# ClusterCleanupPolicy (+ GlobalContextEntry) that runs every minute and deletes -# per-project SCCs whose source namespace is gone or was recreated with a -# different UID, so generated SCCs do not accumulate over the cluster's lifetime. -# Requires the Kyverno cleanup controller to be enabled. -echo "🧹 Installing orphaned-SCC cleanup policy..." -KYVERNO_CLEANUP_RBAC_FILE="${EXTRA_DIR}/kyverno-cleanup-scc-permissions.yaml" -KYVERNO_CLEANUP_POLICY_FILE="${EXTRA_DIR}/kyverno-cleanup-policy.yaml" -ensure_extra_file "${KYVERNO_CLEANUP_RBAC_FILE}" -ensure_extra_file "${KYVERNO_CLEANUP_POLICY_FILE}" -# RBAC first: the cleanup-controller webhook validates at admission that the -# cleanup controller can delete every kind the policy matches, so without this -# the policy is rejected with "cleanup controller has no permission to delete -# kind SecurityContextConstraints". -ssa_apply < "${KYVERNO_CLEANUP_RBAC_FILE}" -ssa_apply < "${KYVERNO_CLEANUP_POLICY_FILE}" +echo "📦 Installing extra OpenShift Kyverno policies..." +# EXTRA_DIR is defined once near the top (defaults to the extra/ dir beside this script). +KYVERNO_OPENSHIFT_FILE="${EXTRA_DIR}/05-kyverno.yaml" +ensure_extra_file "${KYVERNO_OPENSHIFT_FILE}" +sed "s||${DOMAIN}|g" "${KYVERNO_OPENSHIFT_FILE}" | ssa_apply echo "✅ Extra OpenShift Kyverno policies installed" echo "" @@ -702,7 +820,7 @@ step "Workspace StorageClasses (multinode, mlstorage)" # Do not size, schedule or promise workloads on the assumption that these are # distinct backends. Before adding real shared storage (NFS/CephFS/ODF), check # that the access-mode policy is scoped to the local-path classes -- see -# extra/local-path-access-mode-scoped.yaml -- or RWX PVCs on the new storage +# extra/04-local-path-access-mode-scoped.yaml -- or RWX PVCs on the new storage # get downgraded too. Note also that accessModes is immutable: PVCs already # converted to RWO must be recreated to move to real RWX storage. echo "📦 Creating storage classes..." @@ -842,7 +960,7 @@ step "Gateway API CRDs" # Several components (openbao-config, otel-lgtm-stack) render HTTPRoute resources # (gateway.networking.k8s.io/v1) that fail to apply unless the Gateway API CRDs # already exist. Envoy Gateway is the Gateway API implementation (see the ENVOY -# GATEWAY section below); its chart bundles both the upstream Gateway API CRDs and +# AI GATEWAY section below); its chart bundles both the upstream Gateway API CRDs and # the envoy-specific CRDs. We install all of them early so those HTTPRoutes apply # cleanly. Idempotent — installed via --server-side so the later controller install # co-owns them without conflict. @@ -1090,17 +1208,178 @@ echo "⏭️ Skipping MetalLB configuration (not applicable on OpenShift)" echo "" # ============================================================================ -# ENVOY GATEWAY — skipped (OpenShift provides its own routing via HAProxy/Istio) +# ENVOY AI GATEWAY # ============================================================================ -step "Envoy Gateway (skipped on OpenShift; creating namespaces)" -# On OpenShift, the Kyverno policy "generate-routes-from-httproutes" (installed -# as part of kyverno-policies/base) automatically converts Gateway API HTTPRoute -# resources into OpenShift Routes, so components can create HTTPRoutes as normal -# and OpenShift's router will serve them without a separate Gateway controller. -echo "⏭️ Skipping Envoy Gateway installation (OpenShift: using native routing)" -# FIXME: Still create the namespaces since other components reference them -kubectl create namespace envoy-gateway-system --dry-run=client -o yaml | kubectl apply -f - +step "Envoy AI Gateway (https://${AI_HOST})" +# Ordinary web traffic keeps using OpenShift's own routing: the Kyverno policy +# "generate-routes-from-httproutes" turns Gateway API HTTPRoutes into OpenShift +# Routes, so AIWB, Keycloak and the rest need no Gateway controller at all, and +# that stays true whether or not this step runs. +# +# Inference is the exception. Every served model has to answer on one shared +# hostname, with the model chosen per request from the x-ai-eg-backend and +# x-ai-eg-model headers, and an OpenShift Route cannot match on headers — only +# on host and path. So a real Gateway API implementation has to serve +# ${AI_HOST}, and this step installs exactly enough of one to do that. +# +# WHAT IS DELIBERATELY NOT INSTALLED +# - Gateway API CRDs. The ingress-operator owns them and already serves the +# bundle the chart carries; applying them would only take field ownership. +# - The `https` apps gateway from envoy-gateway-config. HAProxy is the front +# door here, and a second one is what would actually collide with cluster +# ingress. Its TLSRoute is replaced by the passthrough Route in +# extra/06-ai-gateway.yaml. +# +# ROLLBACK: `kubectl delete route ai-gateway -n envoy-gateway-system` removes +# the only object in cluster ingress. Everything else is inert without it. + +# cluster-auth is unrelated to the gateway but has always been created here. kubectl create namespace cluster-auth --dry-run=client -o yaml | kubectl apply -f - + +if [ "${IS_OPENSHIFT}" != "true" ]; then + # Everything below needs an OpenShift Route to be reachable and an OpenShift + # SCC to get its pods admitted, so there is nothing useful to install here. + # On RKE2 the equivalent is cluster-forge's own envoy-gateway-config, which + # puts the AI listener behind the `https` apps gateway instead. + echo "⏭️ Skipping Envoy AI Gateway (not an OpenShift cluster)" + # Other components reference this namespace, so create it regardless. + kubectl create namespace envoy-gateway-system --dry-run=client -o yaml | kubectl apply -f - +else + AI_GW_MANIFEST="${EXTRA_DIR}/06-ai-gateway.yaml" + AI_GW_VALUES="${EXTRA_DIR}/07-ai-gateway-values.yaml" + ensure_extra_file "${AI_GW_MANIFEST}" + ensure_extra_file "${AI_GW_VALUES}" + + # Namespaces, SCC, RBAC and the Route. Admission rules have to exist before + # anything can schedule a pod: the SCC in this manifest is what gets the Envoy + # pods past admission at all, and applying it after the Deployments would + # leave them at FailedCreate until something happened to retry them. + echo "📦 Namespaces, SCC, RBAC and Route..." + sed "s|\${DOMAIN}|${DOMAIN}|g" "${AI_GW_MANIFEST}" | ssa_apply + + # The Route is TLS-passthrough, so HAProxy presents no certificate of its own + # and this secret is what clients actually see. Rather than mint a new one, + # copy the certificate the router already serves: it is a wildcard for + # *.${DOMAIN}, so it covers ${AI_HOST} with no extra client trust config. + # + # This is a COPY, not a reference. When the ingress-operator rotates the + # original the copy goes stale, and the symptom is a TLS error on ${AI_HOST} + # alone, with every other hostname fine. Re-running this script refreshes it. + CERT_SRC=$(kubectl get ingresscontroller default -n openshift-ingress-operator \ + -o jsonpath='{.spec.defaultCertificate.name}' 2>/dev/null || true) + CERT_SRC="${CERT_SRC:-router-certs-default}" + echo "🔐 Copying ${CERT_SRC} (openshift-ingress) -> cluster-tls (envoy-gateway-system)..." + AI_GW_CERTDIR=$(mktemp -d) + kubectl get secret "${CERT_SRC}" -n openshift-ingress -o jsonpath='{.data.tls\.crt}' \ + | base64 -d > "${AI_GW_CERTDIR}/tls.crt" + kubectl get secret "${CERT_SRC}" -n openshift-ingress -o jsonpath='{.data.tls\.key}' \ + | base64 -d > "${AI_GW_CERTDIR}/tls.key" + if [ ! -s "${AI_GW_CERTDIR}/tls.crt" ] || [ ! -s "${AI_GW_CERTDIR}/tls.key" ]; then + rm -rf "${AI_GW_CERTDIR}" + echo "❌ Could not read tls.crt/tls.key from secret ${CERT_SRC} in openshift-ingress" >&2 + exit 1 + fi + if command -v openssl >/dev/null 2>&1; then + echo " expires : $(openssl x509 -in "${AI_GW_CERTDIR}/tls.crt" -noout -enddate 2>/dev/null | sed 's/^notAfter=//')" + fi + kubectl create secret tls cluster-tls -n envoy-gateway-system \ + --cert="${AI_GW_CERTDIR}/tls.crt" --key="${AI_GW_CERTDIR}/tls.key" \ + --dry-run=client -o yaml | ssa_apply + rm -rf "${AI_GW_CERTDIR}" + + # Only crds/generated — the chart also ships crds/gatewayapi-crds.yaml, which + # is skipped here for the reason given above. + echo "📦 Envoy Gateway CRDs (Gateway API CRDs skipped — owned by ingress-operator)..." + cat "${SOURCES_DIR}/envoy-gateway/v1.7.1"/crds/generated/*.yaml | ssa_apply + echo "📦 InferencePool CRDs..." + retry helm template inference-extension-crds "${SOURCES_DIR}/inference-extension-crds/v1.5.0" \ + --namespace envoy-ai-gateway-system --include-crds | ssa_apply + echo "📦 Envoy AI Gateway CRDs..." + retry helm template envoy-ai-gateway-crds "${SOURCES_DIR}/envoy-ai-gateway-crds/v0.6.0" \ + --namespace envoy-ai-gateway-system --include-crds | ssa_apply + + # The values file carries the extensionManager block verbatim from + # cluster-forge root/values.yaml. That is what teaches Envoy Gateway to hand + # AIGatewayRoute, AIServiceBackend and InferencePool to the AI controller for + # translation; without it those resources apply cleanly and do nothing. + echo "🚀 Installing Envoy Gateway control plane..." + retry helm template envoy-gateway "${SOURCES_DIR}/envoy-gateway/v1.7.1" \ + --namespace envoy-gateway-system \ + -f "${AI_GW_VALUES}" | ssa_apply + + echo "🚀 Installing Envoy AI Gateway controller..." + retry helm template envoy-ai-gateway "${SOURCES_DIR}/envoy-ai-gateway/v0.6.0" \ + --namespace envoy-ai-gateway-system \ + --set controller.mcp.sessionEncryption.seed="${AI_GATEWAY_MCP_SEED}" | ssa_apply + + echo "⏳ Waiting for the control planes to become available..." + kubectl wait --for=condition=available --timeout=300s \ + deployment/envoy-gateway -n envoy-gateway-system 2>/dev/null \ + || echo "⚠️ envoy-gateway not ready yet; continuing" + kubectl wait --for=condition=available --timeout=300s \ + deployment/ai-gateway-controller -n envoy-ai-gateway-system 2>/dev/null \ + || echo "⚠️ ai-gateway controller not ready yet; continuing" + + # Render only the six templates that apply on OpenShift. The rest of the + # chart targets the apps gateway that HAProxy replaces here, RKE2's CoreDNS + # layout, or the gRPC ext-auth on port 50051 that the cluster-auth shim does + # not expose (it is REST on 8081). + echo "🚀 Installing gateway configuration..." + retry helm template envoy-gateway-config "${SOURCES_DIR}/envoy-gateway-config" \ + --namespace envoy-gateway-system \ + --set domain="${DOMAIN}" \ + --set aiGateway.enabled=true \ + --set aiGateway.routeHostname="${AI_HOST}" \ + -s templates/gateway-class.yaml \ + -s templates/gateway-config-ai.yaml \ + -s templates/ai-gateway.yaml \ + -s templates/ai-gateway-proxy-config.yaml \ + -s templates/ai-gateway-service.yaml \ + -s templates/envoy-extension-policy-model-header.yaml \ + | ssa_apply + + echo "⏳ Waiting for the gateway to be programmed..." + kubectl wait --for=condition=Programmed --timeout=300s \ + gateway/ai-gateway -n envoy-gateway-system 2>/dev/null \ + || echo "⚠️ Gateway not Programmed yet — check the envoy-gateway controller logs" + + # The envoy-ai-gateway chart mints a fresh self-signed certificate on every + # render, so each run rotates the pod-mutating webhook's keypair and caBundle + # together. The controller normally reloads the new cert, but if it ever did + # not, the webhook has failurePolicy: Fail on pod CREATE and would block the + # Envoy data plane from ever starting again — while leaving every other + # workload untouched, since its objectSelector only matches + # app.kubernetes.io/managed-by: envoy-gateway. Cheap to verify, expensive to + # discover by accident. + probe_ai_gateway_webhook() { + kubectl apply --dry-run=server -f - >/dev/null 2>&1 <<'PROBE' +apiVersion: v1 +kind: Pod +metadata: + name: ai-gateway-webhook-probe + namespace: envoy-gateway-system + labels: + app.kubernetes.io/managed-by: envoy-gateway +spec: + containers: + - name: probe + image: registry.access.redhat.com/ubi9/ubi-minimal:latest + command: ["sleep", "1"] +PROBE + } + if probe_ai_gateway_webhook; then + echo "✅ Pod-mutating webhook healthy" + else + echo "⚠️ Pod-mutating webhook is rejecting pods; restarting the AI controller to reload its certificate" + kubectl rollout restart deploy/ai-gateway-controller -n envoy-ai-gateway-system >/dev/null 2>&1 || true + kubectl rollout status deploy/ai-gateway-controller -n envoy-ai-gateway-system --timeout=180s >/dev/null 2>&1 || true + probe_ai_gateway_webhook \ + && echo "✅ Webhook healthy after restart" \ + || echo "❌ Webhook still rejecting pods — the Envoy data plane cannot be recreated until this is fixed" + fi + + echo "✅ Envoy AI Gateway installed (https://${AI_HOST})" +fi echo "" # ============================================================================ @@ -1358,7 +1637,7 @@ if kubectl get crd nodefeaturerules.nfd.k8s-sigs.io >/dev/null 2>&1; then if [ "${AMD_HW_NODES}" != "0" ] && [ "${AMD_LABELLED_NODES}" = "0" ]; then echo "⚠️ AMD GPU hardware detected but no node has feature.node.kubernetes.io/amd-gpu=true" echo "📦 Applying NodeFeatureRule to label AMD GPU nodes..." - NFR_FILE="${EXTRA_DIR}/amd-gpu-nodefeaturerule.yaml" + NFR_FILE="${EXTRA_DIR}/08-amd-gpu-nodefeaturerule.yaml" ensure_extra_file "${NFR_FILE}" retry kubectl apply --request-timeout="${KUBECTL_REQUEST_TIMEOUT}" -f "${NFR_FILE}" echo "⏳ Waiting for nodes to receive the amd-gpu label..." @@ -1386,7 +1665,7 @@ step "AIM Engine (controller + CRDs)" echo "📦 Installing AIM Engine CRDs..." kubectl create namespace aim-system --dry-run=client -o yaml | kubectl apply -f - #retry kubectl apply --request-timeout="${KUBECTL_REQUEST_TIMEOUT}" -f ${SOURCES_DIR}/aim-engine-crds/0.2.2/crds.yaml --namespace aim-system --recursive -retry helm template aim-engine-crds oci://registry-1.docker.io/amdenterpriseai/aim-engine-crds-chart --version 0.2.5 --namespace aim-system | ssa_apply +retry helm template aim-engine-crds oci://registry-1.docker.io/amdenterpriseai/aim-engine-crds-chart --version "$(cf_app_version aim-engine-crds)" --namespace aim-system | ssa_apply # Stage 2: Install AIM Engine operator @@ -1394,7 +1673,7 @@ echo "📦 Installing AIM Engine operator..." # Routing disabled on OpenShift: enabling HTTPRoute routing causes Istio # (pilot-discovery) to fight over the routes, leaving AIMServices stuck in Starting. # The Kyverno "generate-routes-from-httproutes" policy handles exposure via OpenShift Routes. -helm template aim-engine oci://registry-1.docker.io/amdenterpriseai/aim-engine-chart --version 0.2.5 \ +helm template aim-engine oci://registry-1.docker.io/amdenterpriseai/aim-engine-chart --version "$(cf_app_version aim-engine)" \ --namespace aim-system \ --set clusterRuntimeConfig.enable=true \ --set clusterRuntimeConfig.spec.routing.enabled=false \ @@ -1598,7 +1877,7 @@ if [[ "${PLUGGABLE_DB}" != true ]]; then echo " 📦 Installing AIWB database cluster (${CNPG_INSTANCES} instance(s))..." retry helm template aiwb-infra-cnpg \ - oci://registry-1.docker.io/amdenterpriseai/aiwb-cnpg-chart --version 2.0.0 \ + oci://registry-1.docker.io/amdenterpriseai/aiwb-cnpg-chart --version "$(cf_app_version aiwb-infra-cnpg)" \ --set instances=${CNPG_INSTANCES} \ --set username=${AIWB_DB_USER} \ --set storage.storageClass=${DEFAULT_STORAGE_CLASS_NAME} \ @@ -1930,7 +2209,7 @@ if [[ "${PLUGGABLE_DB}" == true ]]; then fi echo "🚀 Installing AIWB application..." -helm template aiwb oci://registry-1.docker.io/amdenterpriseai/aiwb-chart --version 2.0.0 \ +helm template aiwb oci://registry-1.docker.io/amdenterpriseai/aiwb-chart --version "$(cf_app_version aiwb)" \ --namespace aiwb \ --set standAloneMode=true \ --set appDomain="${DOMAIN}" \ @@ -1957,7 +2236,7 @@ echo "" step "AI Gateway Discovery" echo "📦 Installing AI Gateway Discovery..." kubectl create namespace ai-gateway-system --dry-run=client -o yaml | kubectl apply -f - -retry helm template ai-gateway-discovery oci://registry-1.docker.io/amdenterpriseai/ai-gateway-discovery-chart --version 2.0.0 \ +retry helm template ai-gateway-discovery oci://registry-1.docker.io/amdenterpriseai/ai-gateway-discovery-chart --version "$(cf_app_version ai-gateway-discovery)" \ --namespace ai-gateway-system \ --set controller.gateway.routeHostname=ai.${DOMAIN} \ --set controller.gateway.name=ai-gateway \ @@ -2060,7 +2339,7 @@ echo "" step "Kaiwo CRDs" echo "📦 Installing Kaiwo CRD..." #kubectl create namespace kaiwo --dry-run=client -o yaml | kubectl apply -f - -retry helm template kaiwo-crds oci://ghcr.io/silogen/kaiwo-crds-chart --version v0.2.1 \ +retry helm template kaiwo-crds oci://ghcr.io/silogen/kaiwo-crds-chart --version "$(cf_app_version kaiwo-crds)" \ --namespace kaiwo-system \ | ssa_apply @@ -2070,7 +2349,7 @@ retry helm template kaiwo-crds oci://ghcr.io/silogen/kaiwo-crds-chart --version step "Kaiwo " echo "📦 Installing Kaiwo..." -helm template kaiwo oci://ghcr.io/silogen/kaiwo-operator-chart --version v0.2.1 \ +helm template kaiwo oci://ghcr.io/silogen/kaiwo-operator-chart --version "$(cf_app_version kaiwo)" \ --namespace kaiwo-system \ | ssa_apply @@ -2089,7 +2368,7 @@ for f in ${SOURCES_DIR}/kaiwo-config/*.yaml; do echo "---"; cat "$f"; done \ # them as the final step once all Services exist. # ============================================================================ step "Apply OpenShift Routes (AIWB UI/API + Keycloak)" -ROUTES_FILE="${EXTRA_DIR}/routes.yaml" +ROUTES_FILE="${EXTRA_DIR}/09-routes.yaml" echo "🌐 Applying OpenShift Routes..." ensure_extra_file "${ROUTES_FILE}" # Substitute the ${DOMAIN} placeholder in the Route host fields with the cluster's @@ -2149,6 +2428,27 @@ else echo " Ensure DNS points aiwbui.${DOMAIN}, aiwbapi.${DOMAIN}, and kc.${DOMAIN} to ${GATEWAY_IP}" fi echo "" + +if [ "${IS_OPENSHIFT}" = "true" ]; then + echo "🤖 Inference endpoint (all served models share this hostname):" + echo " https://${AI_HOST}/v1/chat/completions" + echo "" + echo " Which model answers is decided by the headers, not the path or host." + echo " -k is needed while the router still serves a self-signed certificate." + echo "" + echo " curl -k -X POST https://${AI_HOST}/v1/chat/completions \\" + echo " -H 'Content-Type: application/json' \\" + echo " -H 'x-ai-eg-backend: ' \\" + echo " -H 'x-ai-eg-model: ' \\" + echo " -d '{\"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}],\"stream\":false}'" + echo "" + echo " Routes appear on their own as models are deployed; the UUID is the" + echo " InferenceService workload-id label:" + echo " kubectl get inferenceservice -A -o custom-columns=\\" + echo " 'NAME:.metadata.name,UUID:.metadata.labels.airm\\.silogen\\.ai/workload-id'" + echo " kubectl get aigatewayroute,aiservicebackend -A" + echo "" +fi echo "💡 Keycloak Admin Credentials:" echo " Username: silogen-admin" echo " Password: placeholder" @@ -2190,3 +2490,52 @@ echo "✅ Cleanup complete" echo "" echo "✅ Deploy complete" + +# TODO: Missing apps??? +# aiwb-infra-external-secrets + + +# kyverno-policies-storage-local-path DONE???? +# cluster-auth-config NOT IN OPENSHIFT??? + + # aiwb-infra-external-secrets: + # repoURL: "{{ .Values.ociRegistry.dockerHub }}" + # repoVersion: "2.0.0" + # chart: "aiwb-external-secrets-chart" + # namespace: aiwb + # syncWave: -20 + # valuesFile: values.yaml + # ignoreDifferences: + # - group: external-secrets.io + # kind: ExternalSecret + # jqPathExpressions: + # - ".spec.data[].remoteRef.conversionStrategy" + # - ".spec.data[].remoteRef.decodingStrategy" + # - ".spec.data[].remoteRef.metadataPolicy" + + # kaiwo-crds: + # repoURL: "{{ .Values.ociRegistry.ghcr }}" + # repoVersion: "v0.2.1" + # chart: "kaiwo-crds-chart" + # namespace: kaiwo + # kaiwo: + # repoURL: "{{ .Values.ociRegistry.ghcr }}" + # repoVersion: "v0.2.1" + # chart: "kaiwo-operator-chart" + # namespace: kaiwo-system + # syncWave: -10 + # kaiwo-config: + # ignoreDifferences: + # - group: external-secrets.io + # jqPathExpressions: + # - ".spec.data[].remoteRef.conversionStrategy" + # - ".spec.data[].remoteRef.decodingStrategy" + # - ".spec.data[].remoteRef.metadataPolicy" + # kind: ExternalSecret + # - group: "" + # jsonPointers: + # - /spec/accessModes + # kind: "PersistentVolumeClaim" + # namespace: kaiwo-system + # path: kaiwo-config + # syncWave: 0