refactor: offload DP/DRA management to KMM

[abyrne55]

This PR replaces the operator's direct DP/DRA DaemonSet management with a single KMM (Kernel Module Management) Module CR. Two controllers collapse into one, and the operator no longer needs to build DaemonSets, manage RBAC, handle pod readiness, or construct OpenShift SCCs at runtime.

	Before	After
Controllers	2 (DevicePluginReconciler, DRAReconciler)	1 (KMMReconciler)
K8s objects managed at runtime	DaemonSets, ServiceAccounts, ClusterRoles, ClusterRoleBindings, DeviceClasses, AdmissionPolicies, SCCs	1 Module CR
DP/DRA embedded YAML + loaders	9 files	0

Existing ClusterPolicy specs work unchanged — KMM is used transparently under the hood.

Beyond simplifying what we already have, KMM also brings capabilities we'd otherwise need to build: OOT kernel module loading (see new CRD field below), kernel-version-aware rollouts, ordered upgrades, pre-flight validation, and build/sign integration. Ordered upgrades and pre-flight validation apply immediately for OOT driver modules; build/sign integration is available when needed.

How it works

A new KMMReconciler sub-controller replaces DevicePluginReconciler and DRAReconciler in the reconcile loop. It uses controllerutil.CreateOrPatch to manage the Module CR idempotently, with SetControllerReference so the Module is garbage-collected when the ClusterPolicy is deleted.

DRA RBAC (ServiceAccount, ClusterRole, ClusterRoleBinding), OpenShift SCCs, and ValidatingAdmissionPolicy resources have been moved out of operator-managed code and into deployment tooling (Helm templates and kustomize overlays).

KMM availability is detected at startup via API group discovery (kmm.sigs.x-k8s.io), same pattern as DRAEnable. If the CRD isn't installed, the operator starts normally and reports an error in ClusterPolicy status.

New CRD field: kernelModule

spec:
  kernelModule:            # optional; omit for in-tree driver
    moduleName: "xe"       # kernel module to load
    image: "registry.example.com/xe-driver:1.0"  # default/fallback image

When kernelModule is set, KMM loads the specified OOT driver module on each node. When omitted, the in-tree kernel driver is used.

Multi-kernel support

kernelMappings maps kernel version patterns to container images, enabling multi-kernel clusters:

spec:
  kernelModule:
    moduleName: xe
    kernelMappings:
    - regexp: "^5\\.14\\.0-.*\\.el9.*\\.x86_64$"
      containerImage: registry.example.com/xe-rhel9:1.0
    - regexp: "^6\\.12\\..*"
      containerImage: registry.example.com/xe-rhel10:1.0

When kernelMappings is empty, a single wildcard mapping (^.+$) is generated from image. When kernelMappings is set, image serves as a KMM-level fallback for mappings that omit containerImage.

In-cluster builds

Each kernel mapping can include a build spec for cluster-time driver image building via KMM:

    kernelMappings:
    - regexp: "^5\\.14\\..*"
      build:
        dockerfileConfigMap:
          name: xe-dockerfile
        buildArgs:
        - name: XE_TAG
          value: v1.0
        secrets:
        - name: private-repo

Additional fields

Field	Description
inTreeModulesToRemove	In-tree modules to unload before OOT insertion. moduleName always included implicitly.
modulesLoadingOrder	Softdep-style loading order for multi-module drivers (first = moduleName, >=2 entries).
firmwarePath	In-container path for firmware files (maps to KMM ModprobeSpec.FirmwarePath).
skipTLSVerify	Disable TLS verification for OOT image registry (maps to KMM RegistryTLS).

KMM status

ClusterPolicyStatus.kmmStatus reports module loader availability (available/desired), visible via kubectl get clusterpolicy as a new KMM column. Shows N/A when kernelModule is nil (in-tree mode).

Mapping to KMM Module CR

KernelModuleSpec	KMM Module CR
ModuleName	ModprobeSpec.ModuleName
Image	ModuleLoaderContainerSpec.ContainerImage (fallback)
KernelMappings[]	ModuleLoaderContainerSpec.KernelMappings[]
InTreeModulesToRemove	ModuleLoaderContainerSpec.InTreeModulesToRemove
ModulesLoadingOrder	ModprobeSpec.ModulesLoadingOrder
FirmwarePath	ModprobeSpec.FirmwarePath
SkipTLSVerify	RegistryTLS.InsecureSkipTLSVerify

Aligned with tkatila's DriverXeSpec API sketch. Genericized Xe-specific fields (e.g., XeTag/XeSHA → BuildArgs). Skipped fields already covered elsewhere (NodeSelector, PullSecret, Enable) and Xe-specific enums (XeOSTarget). No Sign spec yet.

Known limitations

• In-cluster build untested. Build spec support is implemented and unit-tested but hasn't been validated end-to-end.
• DP/DRA containers run privileged. KMM unconditionally sets privileged: true on DP and DRA containers. The old DP ran unprivileged with a hardened security context. There is no override via the Module CR — this would require an upstream KMM change (optional SecurityContext on CommonContainerSpec). Implementing PR Support configurable SecurityContext for DevicePlugin and DRA containers kubernetes-sigs/kernel-module-management#1323 under review.
• DRA StartupProbe lost. KMM adds a hardcoded LivenessProbe but no StartupProbe. The old operator configured a StartupProbe with a 10-minute grace window for slow DRA driver startup. CommonContainerSpec has no probe fields, so this requires an upstream KMM enhancement. Implementing PR Add configurable StartupProbe and LivenessProbe to Module CR kubernetes-sigs/kernel-module-management#1322 under review.
• Upgrade path. Existing deployments will have orphaned cluster-scoped resources (ClusterRoles, ClusterRoleBindings, SCCs, DeviceClasses, ValidatingAdmissionPolicies) from the pre-KMM operator that require manual cleanup.

Test results

Tested on OCP 4.22.2 (SNO) with two Intel Arc Pro B70 GPUs. KMM and NFD pre-installed.

• DP mode with in-tree driver: Module CR created, KMM spun up DP DaemonSet, GPU devices registered, test pod scheduled and saw /dev/dri devices
• DRA mode with in-tree driver: Module CR created, KMM spun up DRA DaemonSet, DeviceClasses created, GPU devices visible in ResourceSlices, test pod with DRA ResourceClaim scheduled and saw /dev/dri devices
• DRA mode with OOT driver (simple moduleName + image): KMM loaded OOT xe module, DRA ResourceSlices populated, test pod saw GPUs
• XPU Manager runs alongside KMM-managed DRA
• ClusterPolicy deletion cleans up Module CR (DaemonSet garbage-collected by KMM)
• make test passes (all specs including expanded KMM controller tests)

This PR was written in part with the assistance of generative AI.

[tkatila]

Hi @abyrne55 ! Thanks for the PoC!

• K8s object creation (serviceaccount, SCCs, etc.) is moved from operator runtime to operator deployment time, what is the reasoning for this move?
  • While it can have security benefits (less access rights in the operator), it limits what the operator can do.
  • It also forces to use the "is openshift" variable in the chart.
  • https://operatorframework.io/operator-capabilities/ would suggest that operator should be capable of creating objects, but I'm not sure if there is general consensus on it.
• Was there a specific KMM version required for this to work? I just applied KMM via the kubectl oneliner from the repo. Seems to be "latest" image. I'm getting this error with DRA:

E0722 06:33:59.300935       1 clusterpolicy_controller.go:234] Return sub-controller errorfailed to reconcile KMM Module gpu-policy-gpu for ClusterPolicy gpu-policy: admission webhook "vmodule.kb.io" denied the request: failed to validate DRA volumes: spec.dra.volumes[0]: hostPath "/etc/cdi" is not allowed; only /dev, /sys, /var, /opt and /run paths are permitted

• And this with DP (in KMM):

E0722 06:31:42.060239       1 controller.go:347] "Reconciler error" err="could handle device plugin: DaemonSet.apps \"g
pu-policy-gpu-device-plugin-l9t68\" is invalid: spec.template.spec.containers[0].volumeMounts[3].mountPath: Invalid value: \"/var/lib/kubelet/device-plugins\": must be unique" logger="kmm" controller="DevicePluginReconciler" controllerGroup="kmm.sigs.x-k8s.io" controllerKind="Module" Module="intel-gpu-base-operator/gpu-policy-gpu" namespace="intel-gpu-base-operator" name="gpu-policy-gpu" reconcileID="57b85f12-1653-490f-b814-05f0da238e78"

• I see KMM has received a helm chart (Adding helm deployment kubernetes-sigs/kernel-module-management#1309). Is there a timeline for the first release with it?

I'm still thinking if I'd like to have DP/DRA deployment also in the base-operator side. For anyone not wanting to deploy KMM to their cluster. Both methods could use the same building blocks so there wouldn't be too much overhead. More testing though.

[abyrne55]

@tkatila thanks for taking a look!

My latest push should resolve the hostPath and "duplicate mount" bugs: the /etc/cdi volume was removed since KMM sets CDI_ROOT=/var/run/cdi, and the duplicate /var/lib/kubelet/device-plugins mount was dropped since KMM auto-adds it.

Re. the KMM version, I've been using "bleeding edge" builds from rh-ecosystem-edge/kernel-module-management:main while we wait for the next KMM y-stream release (release-2.7 branch), which fixes a couple DRA-related bugs that impeded my earlier testing. If you're only testing with the device plugin, then you should be okay using the latest stable KMM version.

On the K8s object creation move: the main motivation was reducing the operator's RBAC footprint. Agreed it's a tradeoff: happy to discuss where to draw the line.

Re. keeping a non-KMM path: let's discuss. I think that might negate a lot of the maintainability benefits of this PR.

[tkatila]

My latest push should resolve the hostPath and "duplicate mount" bugs: the /etc/cdi volume was removed since KMM sets CDI_ROOT=/var/run/cdi, and the duplicate /var/lib/kubelet/device-plugins mount was dropped since KMM auto-adds it.
Re. the KMM version, I've been using "bleeding edge" builds from rh-ecosystem-edge/kernel-module-management:main while we wait for the next KMM y-stream release (release-2.7 branch), which fixes a couple DRA-related bugs that impeded my earlier testing. If you're only testing with the device plugin, then you should be okay using the latest stable KMM version.

Roger. I'll try the latest patch set and KMMO build tomorrow.

On the K8s object creation move: the main motivation was reducing the operator's RBAC footprint. Agreed it's a tradeoff: happy to discuss where to draw the line.

Right. I do like the reduced footprint. I'll ask internally if we have any preference on the direction.

Re. keeping a non-KMM path: let's discuss. I think that might negate a lot of the maintainability benefits of this PR.

Can you help me understand what benefits KMMO's DP/DRA handling has compared to the current state? AFAIK, main benefits from KMMO are related to the KMD handling, and the DP/DRA handling is more like helper functionality. Are kernel-version-aware rollouts, ordered upgrades, or pre-flight validation functionalities tied to DP/DRA being part of the Module CR?

I see moving the DP/DRA handling to KMMO as a risk mostly. It introduces unknown delays to our internal timelines if there are issues/missing functionality with KMMO (like your two PRs related to security and probes). And without KMMO, the ClusterPolicy CRD is effectively broken. KMMO dependency can be made more visible via helm chart prehooks, CRD's webhooks, or I think even with operator bundle's dependencies, but I feel KMMO is a dependency for OoT KMD handling, not a core component for the base operator to function.

[abyrne55]

Thanks for your feedback @tkatila!

Can you help me understand what benefits KMMO's DP/DRA handling has compared to the current state?

The main benefit is less reconciler code to maintain within gpu-base-operator: KMM owns the DP/DRA DaemonSet lifecycle instead of the operator managing it directly. The Module CR also provides a single point of observability for the full driver+DP/DRA stack, though that's an admittedly minor benefit.

Are kernel-version-aware rollouts, ordered upgrades, or pre-flight validation functionalities tied to DP/DRA being part of the Module CR?

For OOT scenarios: partially. KMM's pre-flight validation is only concerned with OOT KMD images, and it doesn't currently care about DP/DRA. Ordered upgrades do include DP/DRA: KMM's version label state machine enforces old DP/DRA drain → old KMD unload → new KMD load → new DP/DRA deploy. And DP/DRA scheduling is gated on the kmm.node.kubernetes.io/<ns>.<name>.ready node label, which is only set once the module loads successfully.

For in-tree scenarios: no, as these KMM features don't engage without an OOT KMD image configured.

I see moving the DP/DRA handling to KMMO as a risk mostly. It introduces unknown delays to our internal timelines if there are issues/missing functionality with KMMO (like your two PRs related to security and probes). And without KMMO, the ClusterPolicy CRD is effectively broken.

These are good points, and your concerns re. timeline coupling are especially valid.

I feel KMMO is a dependency for OoT KMD handling, not a core component for the base operator to function.

Given this discussion, I think this could be a reasonable design. I'll sketch out a version of this PR that illustrates a KMM-optional path to ensure we're all on the same page. I'll also discuss this approach internally to ensure we're not missing anything.

[tkatila]

For OOT scenarios: partially. KMM's pre-flight validation is only concerned with OOT KMD images, and it doesn't currently care about DP/DRA. Ordered upgrades do include DP/DRA: KMM's version label state machine enforces old DP/DRA drain → old KMD unload → new KMD load → new DP/DRA deploy. And DP/DRA scheduling is gated on the kmm.node.kubernetes.io/..ready node label, which is only set once the module loads successfully.

DP/DRA should handle KMD coming and going. As they are not accessing the drm devices themselves, they should be fine running through KMD change.

But XPUMD is different. That will probably crash when KMD vanishes underneath it. Maybe the ready label should be used for its DaemonSet.

Given this discussion, I think this could be a reasonable design. I'll sketch out a version of this PR that illustrates a KMM-optional path to ensure we're all on the same page. I'll also discuss this approach internally to ensure we're not missing anything.

That would be my preference. I think this PR shows that it's possible to move the DP/DRA handling to KMMO also later, if there are more pressing reasons for it.

[abyrne55]

Closing in favor of #79. We may revisit handing off DRA/DP handling to KMM in the future