EAI-5958: Stamp workload_id on per-pod CPU/memory metrics

[johnl-amd]

Summary

• Adds a dedicated kubeletstats daemonset collector (otel-collector-metrics-kubelet) to the otel-lgtm-stack chart that emits per-pod CPU and memory metrics labeled with workload_id (plus project_id, pod, namespace, container).
• Enables per-workload CPU/system-memory observability for EPYC (CPU-only) AIMs in the AIWB dashboard, scoped the same stable way GPU/VRAM metrics already are.

Why

EPYC-based AIMs (parent initiative EAI-2244) hold model weights in RAM, so the AIWB dashboard needs CPU + system-memory panels alongside the existing GPU/VRAM panels. The AIWB metrics backend scopes every panel by a workload_id="<AIM-uuid>" label. GPU metrics carry that label (stamped by the device-metrics-exporter); vLLM metrics carry it (vLLM collector relabel). Container CPU/memory metrics do not.

Root cause of the gap

otel-collector-metrics-k8s scrapes cAdvisor via a role: node scrape (/metrics/cadvisor). Those metrics are node-scoped, so pod/namespace arrive as datapoint labels. The k8sattributes processor associates pods at the resource level, so it cannot enrich node-scoped cAdvisor series — leaving container_cpu_usage_seconds_total / container_memory_working_set_bytes without workload_id.

Approach

Use the kubeletstats receiver instead, which emits pod/container-scoped metrics with k8s.pod.* as resource attributes:

1. kubeletstats (daemonset, per-node kubelet at :10250) → pod + container metric groups.
2. k8sattributes looks up the pod and extracts airm.silogen.ai/workload-id → workload.id (and project-id).
3. transform/workload-id-labels copies the identity resource attributes down to datapoint attributes, so they survive OTLP→Mimir ingestion as series labels (resource attributes otherwise land in target_info, not on the series).

Reuses the existing otel-collector ServiceAccount — RBAC already grants nodes/stats, nodes/proxy, pods, namespaces, so no RBAC change is needed.

Risk

Low–medium. Additive: a new collector; existing pipelines are untouched. New series (k8s_pod_cpu_*, k8s_pod_memory_*, k8s_container_*) with per-pod cardinality (comparable to existing cAdvisor series). Uses the same collector image and OTLP endpoint as the other collectors.

Test plan / validation status

• Config review against the deployed v1.0.7 collectors and the working k8sattributes/workload-id pattern already used by the logs collector and the vLLM collector.
• Runtime validation deferred — app-dev is currently unreachable and the local kind cluster is degraded (host inotify limit exhausted → kube-proxy/CoreDNS/OTel-operator crashlooping, so the operator webhook can't admit the new collector). Kept as draft until validated on a live cluster.
• On deploy, verify: k8s_pod_cpu_usage{workload_id="<aim-uuid>"} and k8s_pod_memory_working_set_bytes{workload_id="<aim-uuid>"} return series for a labeled workload pod, and that workload_id/namespace/pod appear as labels.

Follow-ups (separate work, not in this PR)

• AIWB backend: add MetricName entries + PromQL builders for CPU usage / CPU availability / system memory.
• AIWB frontend: CPU/memory panels + accelerator-type gating.
• CPU/memory availability denominator sourced from the K8s object (resolved-profile requests/limits) rather than kube-state-metrics.

[johnl-amd]

Superseded by #807, which is the same kubeletstats collector change based on bump_version_envoy_and_ai_gateway (the branch currently on app-dev) instead of main, so it lands on top of the gateway bump without reverting it.