EAI-5958: Add kubeletstats collector to stamp workload_id on pod CPU/mem metrics - #807
Open
johnl-amd wants to merge 31 commits into
Open
EAI-5958: Add kubeletstats collector to stamp workload_id on pod CPU/mem metrics#807johnl-amd wants to merge 31 commits into
johnl-amd wants to merge 31 commits into
Conversation
The v0.6.0 -> v1.0.0 envoy-ai-gateway chart upgrade dropped the ClusterRole/ClusterRoleBinding granting the envoy-gateway service account get/list/watch on aigateway.envoyproxy.io aigatewayroutes and aiservicebackends. These are configured as extensionManager resources, so envoy-gateway watches them directly and was denied, causing its informer caches to never sync and the controller to crashloop (0/1). Re-add the template to close the RBAC gap.
v1.0.0's controller fails its gateway reconcile with "found gateway-labeled objects in multiple namespaces" whenever the Gateway lives in the Envoy Gateway namespace (envoy-gateway-system), because getObjectsForGateway does not dedup gw.Namespace against envoyGatewayNamespace. This blocks extproc filter-config generation, so the ai-gateway proxy cannot roll to v1.0.0. Fixed upstream in envoyproxy/ai-gateway#2304 (not yet in a tagged release). Pin controller and extProc images to a main build containing the fix (v1.0.0 + 11 commits). Remove once a release past v1.0.0 ships #2304.
bump_version had the envoy/ai-gateway version bump but not the ai-gateway auth cutover, so ai-gateway was still behind cluster-auth ext_authz — which does not reject the header-less/no-key path, leaving the API-key bypass open (verified live on app-dev: no/bogus key with no x-ai-eg-* headers -> 200). Port the EAI-7304 changes from EAI-7502 (final, post-revert state): - security-policy-ai-gateway-default-deny.yaml: gateway-scoped defaultAction Deny backstop on ai-gateway; per-model apiKeyAuth policies from ai-gateway-discovery override it per-route, so keyless/unmatched requests are denied instead of served. - security-policy-extauth.yaml: drop ai-gateway from cluster-auth ext_authz (two gateway-scoped SecurityPolicies cannot target the same Gateway). - values.yaml: document the cutover (gated on aiGateway.enabled). - ai-gateway-proxy-config.yaml / envoy-proxy-access-logs.yaml: attribute aim_service_id via x-ai-eg-backend.
Test the POC's mechanism for closing the header-less API-key bypass: - filterOrder: move lua before api_key_auth (was before ext_authz, a no-op since ext_authz was removed from ai-gateway). - set-model-header-from-body Lua: call clearRouteCache() after setting x-ai-eg-backend/model so the request re-selects the keyed HTTPRoute before api_key_auth evaluates. Empirical trial: cluster-forge routes on backend-UUID (ext_proc-resolved), unlike the POC's model-name routing, so this may not fully close it — testing on app-dev to confirm.
…Cache" This reverts commit 5d93cd0.
Reinstates the reverted POC to close the header-less API-key bypass on the upgraded stack (EG v1.8.1 / AI Gateway v1.0.0). filterOrder moves lua ahead of api_key_auth so set-model-header-from-body derives x-ai-eg-model/backend from the body and clearRouteCache() re-selects the keyed HTTPRoute before per-model apiKeyAuth evaluates. Discovery still emits the SP-covered model-name rule on app-dev, so a model-only body (no backend UUID) lands on an auth-covered route -- the condition that was absent when this was first reverted. To validate on app-dev: confirm lua relocates to filter index 0 (config_dump) and the body-only no/bogus-key path returns 401.
Re-tested live on app-dev (EG v1.8.1 / AI Gateway v1.0.0): the reorder was honored (lua moved ahead of api_key_auth) but the body-only bypass stayed open. Root cause proven via access logs -- ext_proc/aigateway anchors ahead of lua (lua is in afterExtProcFilterPrefixes) and consumes the body, so the Lua's request_handle:body() is empty and it never sets x-ai-eg-model. Config filterOrder cannot place lua ahead of ext_proc. Reverting to baseline before the EnvoyPatchPolicy-based approach (json_to_metadata + header_mutation ahead of api_key_auth, which are NOT in the anchoring prefix list).
Variant A of the header-less API-key bypass fix. Enables enableEnvoyPatchPolicy and injects json_to_metadata + header_mutation at the front of the ai-gateway listener, ahead of api_key_auth. These filters are not in the ai-gateway afterExtProcFilterPrefixes anchor list, so (unlike the reverted Lua POC) they run before ext_proc consumes the body; json_to_metadata reads the body non-destructively so ext_proc still gets it. x-ai-eg-model is set from the body model field before api_key_auth, so a body-only request matches the SP-covered model route and authenticates. Requires an envoy-gateway restart after sync (enableEnvoyPatchPolicy). Open item to validate on app-dev: route re-computation -- if the catch-all route is cached before header_mutation runs, api_key_auth may read stale per-route config; the fallback is Variant B (buffer exception + Lua clearRouteCache).
…te wins Variant A set x-ai-eg-model from the body before api_key_auth, but the route was still the cached catch-all (header_mutation does not clear the route cache), so api_key_auth passed the SP-less route and the body-only bypass stayed 200. Add a tiny clearRouteCache Lua between header_mutation and api_key_auth so api_key_auth re-resolves the route from the now-set x-ai-eg-model header (the SP-covered model route). Named non-prefixed so ai-gateway does not anchor its ext_proc ahead of it.
…idn't apply) Live stats showed json_to_metadata.rq.success (body parsed, metadata set) but api_key_auth.allowed on body-only requests -- i.e. header_mutation's %DYNAMIC_METADATA% value never populated x-ai-eg-model, so api_key_auth still saw the catch-all route. Replace header_mutation with a custom-named Lua that reads ai.body:model from dynamic metadata, sets x-ai-eg-model when absent, and clearRouteCache() so api_key_auth re-resolves onto the SP-covered model route.
…ng phase) json_to_metadata populates dynamic metadata in the data phase, but the Lua and api_key_auth run in the header phase -- so the metadata was never available when the Lua read it, and x-ai-eg-model stayed unset. Switch to a single Lua that reads the body directly (request_handle:body() blocks until the body is buffered, so it works in the header phase). The Lua is ahead of ext_proc (non-prefixed name) so the body is intact; it sets x-ai-eg-model when absent and clearRouteCache so api_key_auth re-resolves onto the SP-covered model route.
The EnvoyPatchPolicy path (json_to_metadata/header_mutation/body-read Lua) was proven a dead end: the body->header transform works (x-ai-eg-model set before api_key_auth, confirmed) but api_key_auth does not honor the per-model SP on a clearRouteCache-re-resolved route, so the body-only bypass stayed 200. Restore the branch to the POC state (filterOrder lua-before-api_key_auth + clearRouteCache) and drop enableEnvoyPatchPolicy + the EnvoyPatchPolicy manifest.
Merge an extAuth block into the gateway-scoped ai-gateway-default-deny policy (Envoy Gateway allows only one SecurityPolicy per Gateway) so header-less inference requests -- standard OpenAI clients that put the model in the body -- are authorized by ai-gateway-discovery's ext_authz handler. bodyToExtAuth forwards the body, Authorization is forwarded in, and x-api-key-id back out for metrics. Add the cross-namespace ReferenceGrant for the discovery Service. On keyed HTTPRoutes the per-model route SecurityPolicy overrides this whole policy (extAuth included), so extAuth only authorizes the header-less path -- a supplement to native per-model apiKeyAuth until upstream ai-gateway supports body-path client auth. Requires the discovery image/chart that exposes the authz port.
The ai-gateway-discovery Service is deployed in the ai-gateway-discovery namespace (ArgoCD destination), not ai-gateway-system. Fix the extAuth backendRef and the ReferenceGrant to target ai-gateway-discovery so the SecurityPolicy is accepted.
b078992 changed the ext_authz backendRef/grant to ai-gateway-discovery to match what was live on app-dev, but that deployment is a test install straight from the core branch (github silogen/core, chart into ai-gateway-discovery), not the canonical path. cluster-forge declares the discovery app in ai-gateway-system (published chart), so the extAuth must target ai-gateway-system.
Hardcoding the discovery namespace was fragile: it's ai-gateway-system canonically but differs per env (app-dev's overlay runs discovery in ai-gateway-discovery), and a mismatch makes the SecurityPolicy fail to attach (extAuth silently not applied -> the body-only bypass returns). Drive the extAuth backendRef + ReferenceGrant namespace from .Values.aiGateway.discoveryNamespace, wired in root/values.yaml to .Values.apps.ai-gateway-discovery.namespace so it always matches where discovery deploys.
…arameter
Go templates can't dot-access a map key containing '-', so
{{ .Values.apps.ai-gateway-discovery.namespace }} failed helm lint. Use
{{ index .Values.apps "ai-gateway-discovery" "namespace" }}.
ai-gateway: wire body-aware ext_authz on the default-deny SecurityPolicy
The v1.0.0 ai-gateway chart ships requestHeaderAttributes and metricsRequestHeaderAttributes as empty strings (v0.6.0 populated them), so the deployment template omits the --metricsRequestHeaderAttributes controller flag. Without it the ext_proc never maps x-api-key-id -> api_key_id (nor x-ai-eg-backend -> aim_service_id) onto the gen_ai_* metrics, leaving those labels empty and breaking per-API-key usage stats in the AIWB UI. Set both attributes in the envoy-ai-gateway valuesObject. aim_service_id tracks x-ai-eg-backend to match the access-log mapping, since the new gateway build no longer emits x-aim-service-id.
Move the requestHeaderAttributes / metricsRequestHeaderAttributes mapping from the root valuesObject override into the vendored v1.0.0 chart default, matching how the v0.6.0 chart shipped them. The v1.0.0 chart defaulted them to "", which dropped the controller flags and blanked api_key_id / aim_service_id on the gen_ai_* metrics — breaking AIWB per-API-key usage stats for any install, not just this one. Keeping it in the chart default (single source of truth) rather than a root override means fresh installs and other environments get correct per-key metrics without extra wiring. aim_service_id maps from x-ai-eg-backend since the current build no longer emits x-aim-service-id. The ai-gateway controller/extProc image stays pinned to 8007dfd (v1.0.0 + 11 commits) because the crash fix #2304 is not yet in any release; v1.0.0 is the latest and reintroduces the reconcile crash. Revisit when a release ships past v1.0.0.
Clean up the branch for review now that the final design has settled (native apiKeyAuth cutover + PR 793 body-aware ext_authz via discovery): - Remove the set-model-header-from-body Lua EnvoyExtensionPolicy and its filterOrder wiring on the ai-gateway EnvoyProxy. It was a superseded POC (repeatedly reverted as ineffective): header-less/body requests are authorized by ai-gateway-discovery's body-aware ext_authz, which parses backend/model from the forwarded body and enforces per-backend/per-model auth. Verified on app-dev that the Lua is not the effective mechanism (wrong-backend requests return discovery's error, not native apiKeyAuth's). - Drop the old vendored charts left by the version bump (envoy-gateway/v1.7.1, envoy-ai-gateway/v0.6.0, envoy-ai-gateway-crds/v0.6.0), now unreferenced by root/values.yaml, and point sbom/components.yaml at the current versions. - Refresh stale comments that referenced the removed lua-before-ext_authz chain. No behavior change intended; the apps-gateway (https) cluster-auth filterOrder is unrelated and left as-is.
Add x-ai-eg-backend to the default-deny SP's headersToBackend so the resolved backend that discovery forwards (core #4309) reaches the ext_proc and populates aim_service_id on the header-less path, where the gateway can't resolve the backend from EDS endpoint metadata. Stacked on the bump branch (that's where the default-deny SP lives); retarget to main after #794 merges.
EAI-7243: forward x-ai-eg-backend for header-less metric attribution
…aim_service_id Add a sanitization-only Lua EnvoyExtensionPolicy (sanitize-backend-header) that drops x-ai-eg-backend unless it is a well-formed workload UUID, so a client- supplied bogus value (e.g. "foobar") is never recorded as the aim_service_id access-log field / gen_ai metric label. Distinct from the removed set-model-header-from-body POC: it does not read the body, set model/backend, route, clear the route cache, or authorize — pure metric hygiene. Removing a non-UUID value is safe for routing (such a value can never match a backend route rule's Exact UUID match). Re-adds a minimal filterOrder (lua before api_key_auth and ext_proc) so the strip precedes header capture; metric coverage is ordering-dependent and must be verified on deploy. Real cross-model attempts (a valid foreign backend UUID) are already denied by native apiKeyAuth via the EAI-7864 routing fix; a valid-format fake UUID is an accepted residual.
…mem metrics Per-pod CPU and memory time-series are needed for EPYC (CPU-only) AIM observability in the AIWB dashboard, scoped the same way GPU/VRAM metrics already are: by a stable workload_id label. cAdvisor metrics (scraped by otel-collector-metrics-k8s via a node-role scrape) are node-scoped, so pod/namespace arrive as datapoint labels and cannot be associated to a pod by the k8sattributes processor, which works at the resource level. As a result container_cpu_usage_seconds_total / container_memory_working_set_bytes carry no workload_id. Add a dedicated daemonset collector using the kubeletstats receiver, which emits pod/container-scoped metrics with k8s.pod.* as resource attributes. k8sattributes then looks up the pod and stamps its airm.silogen.ai/workload-id (and project-id) label; a transform copies the identity resource attributes down to datapoint attributes so they survive OTLP ingestion as series labels (workload_id, project_id, pod, namespace, container). Reuses the existing otel-collector ServiceAccount (RBAC already grants nodes/stats).
The kubeletstats receiver used endpoint https://${env:K8S_NODE_NAME}:10250 with
K8S_NODE_NAME from spec.nodeName. Cluster DNS does not resolve node hostnames, so
every scrape failed with 'no such host' and the collector produced no metrics.
Use status.hostIP (always routable) instead.
…rations - Drop the 'container' kubeletstats metric group: consumers query the k8s_pod_* series, so the container group only added per-container cardinality (overlapping cAdvisor). - Add catch-all tolerations so the daemonset runs on tainted nodes (GPU/inference), collecting per-workload CPU/memory fleet-wide like the node-exporter daemonset.
Add filter.node_from_env_var so each daemonset pod's k8sattributes processor watches only its own node's pods (cost scales with pods-per-node, not pods x nodes). kubeletstats already scrapes only the local node, so local pod metadata is all the enrichment needs. Re-adds a K8S_NODE_NAME env used solely as the local watch-filter key (not for DNS).
Contributor
Author
Why collect CPU/memory this way (kubeletstats + k8sattributes)Recording the design rationale, since the approach isn't obvious. Requirement: AIWB needs per-pod CPU/memory series natively labelled with Why not reuse the existing cAdvisor scrape? We already scrape Why not a query-time join? We could keep cAdvisor and join against Why not source-side stamping (like GPU/vLLM)? GPU works because the device-metrics-exporter knows which workload owns each GPU and labels at emission; vLLM relabels its own metrics. Generic CPU/memory comes from the kernel/cgroups via the kubelet, which is workload-agnostic — there's no app to stamp Chosen approach: Cost & mitigations: this path needs a DaemonSet (each node's kubelet is only reachable locally) and enrichment (hence a pod watch) — costs GPU/vLLM avoid by getting the label at the source. Mitigated here by: node-scoping the Not prescribed by EAI-5958 (the epic specifies the outcome, not the mechanism) — this is the mechanism that keeps CPU/memory consistent with the existing GPU/vLLM metric shape. |
nowycondro
approved these changes
Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
kubeletstatsdaemonset collector (otel-collector-metrics-kubelet) to theotel-lgtm-stackchart that emits per-pod CPU and memory metrics labeled withworkload_id(plusproject_id,pod,namespace,container).Why
EPYC-based AIMs (parent initiative EAI-2244, epic EAI-5958) hold model weights in RAM, so the AIWB dashboard needs CPU + system-memory panels alongside the existing GPU/VRAM panels. The AIWB metrics backend scopes every panel by a
workload_id="<AIM-uuid>"label. GPU metrics carry that label (stamped by the device-metrics-exporter); vLLM metrics carry it (vLLM collector relabel). Container CPU/memory metrics do not.Root cause of the gap
otel-collector-metrics-k8sscrapes cAdvisor via arole: nodescrape (/metrics/cadvisor). Those metrics are node-scoped, sopod/namespacearrive as datapoint labels. Thek8sattributesprocessor associates pods at the resource level, so it cannot enrich node-scoped cAdvisor series — leavingcontainer_cpu_usage_seconds_total/container_memory_working_set_byteswithoutworkload_id.Approach
Use the
kubeletstatsreceiver instead, which emits pod/container-scoped metrics withk8s.pod.*as resource attributes:kubeletstats(daemonset, per-node kubelet at:10250) → pod + container metric groups. The kubelet is addressed by node IP (status.hostIP), not node name — this cluster's CoreDNS doesn't resolve node hostnames, so${env:K8S_NODE_NAME}:10250fails withno such host.k8sattributeslooks up the pod and extractsairm.silogen.ai/workload-id→workload.id(andproject-id).transform/workload-id-labelscopies the identity resource attributes down to datapoint attributes, so they survive OTLP→Mimir ingestion as series labels (resource attributes otherwise land intarget_info, not on the series).Reuses the existing
otel-collectorServiceAccount — RBAC already grantsnodes/stats,nodes/proxy,pods,namespaces, so no RBAC change is needed.Risk
Low–medium. Additive: a new collector; existing pipelines are untouched. New series (
k8s_pod_cpu_*,k8s_pod_memory_*,k8s_container_*) with per-pod cardinality (comparable to existing cAdvisor series). Uses the same collector image and OTLP endpoint as the other collectors.Test plan / validation status — ✅ validated on app-dev (2026-08-12)
Validated end-to-end against existing labeled pods (no EPYC model needed — the collector stamps
workload_idon any pod carryingairm.silogen.ai/workload-id):no such host).k8s_pod_memory_working_set_bytes{workload_id="<aim-uuid>"}returns a series carryingworkload_id,project_id,namespace,podas labels.k8s_pod_cpu_utilization_ratio{workload_id="<aim-uuid>"}returns the pod's CPU ratio.workload_idon their pod memory series.Emitted metric names (note the OTLP→Prometheus unit suffixes, for the AIWB backend follow-up):
k8s_pod_cpu_utilization_ratio(ratio → ×100 for %),k8s_pod_cpu_time_seconds_total(counter)k8s_pod_memory_working_set_bytes,k8s_pod_memory_usage_bytesk8s_pod_memory_available_bytesFollow-ups (separate work, not in this PR — the rest of epic EAI-5958)
MetricNameentries + PromQL builders for CPU usage / CPU availability / system memory, using the actual emitted names above (k8s_pod_cpu_utilization_ratio,k8s_pod_memory_working_set_bytes, …).