Add optional KMM integration for OoT kernel module loading

[abyrne55]

This PR adds a KMMReconciler sub-controller that creates a KMM Module CR when the new ClusterPolicy.spec.kernelModule field is set and KMM is installed. The existing DP and DRA controllers keep their lifecycle logic. KMM handles only out-of-tree kernel module loading (modprobe, kernel mappings, in-cluster builds).

How it works

When ClusterPolicy.spec.kernelModule is set and KMM is installed in the cluster, KMMReconciler creates a KMM Module CR with a moduleLoader spec. The Module CR is owned by the ClusterPolicy and garbage-collected on deletion. A deletion guard skips Module CR removal while GPU ResourceClaims are still allocated.

KMM availability is detected at startup via API group discovery (kmm.sigs.x-k8s.io), same pattern as DRAEnable. If kernelModule is set but KMM isn't installed, the operator reports an error in ClusterPolicy status. If kernelModule is nil, the sub-controller is a no-op.

Downstream DaemonSets (DP, DRA, XPU Manager) are gated on the KMM ready node label (kmm.node.kubernetes.io/<ns>.<module>.ready), so they only schedule once the OoT module is actually loaded on a node. When kernelModule unset, however, their behavior is unchanged.

Relationship to PR #59

#59 proposed replacing the native DP/DRA controllers entirely with KMM. Two controllers collapse into one, and gpu-base-operator stops managing DP/DRA DaemonSets, RBAC, SCCs, and DeviceClasses at runtime. That gives the biggest code reduction (~-3,100 lines) but makes KMM a hard dependency for all users.

This PR takes a more conservative approach: the native controllers stay in place for DP/DRA lifecycle, and a new KMMReconciler runs alongside them solely for OoT kernel module management. KMM is never required, and clusters using in-tree drivers work exactly as before. The trade-off is more total code (+2,456 / -51 lines across 30 files).

kernelModule API

kernelMappings must contain at least 1 entry. Minimal usage: one OoT driver image matched to all kernels:

spec:
  kernelModule:
    kernelMappings:
    - regexp: "^.+$"
      containerImage: registry.example.com/xe-driver:1.0

More realistic usage: different images per kernel version

spec:
  kernelModule:
    moduleName: xe
    kernelMappings:
    - regexp: "^5\\.14\\.0-.*\\.el9.*\\.x86_64$"
      containerImage: registry.example.com/xe-rhel9:1.0
    - regexp: "^6\\.12\\..*"
      containerImage: registry.example.com/xe-rhel10:1.0

In-cluster build mode: useful when driver source is available but pre-built images don't exist for every kernel version in the fleet. KMM checks the target registry first and only triggers a build if the image is missing.

    kernelMappings:
    - regexp: "^5\\.14\\..*"
      build:
        dockerfileConfigMap:
          name: xe-dockerfile
        buildArgs:
        - name: XE_TAG
          value: v1.0
        secrets:
        - name: private-repo

The dockerfileConfigMap references a ConfigMap containing the Dockerfile. buildArgs are passed as build arguments, and secrets are mounted during the build (e.g., for private source repos). Registry auth should use pullSecret on ClusterPolicySpec instead.

Fields

Field	Description
moduleName	Kernel module to load. Defaults to xe.
version	Passed to the KMM container spec to trigger module redeploys on change.
kernelMappings[].regexp	Required. Kernel version pattern for this mapping.
kernelMappings[].containerImage	Pre-built OoT driver image for matching kernels.
kernelMappings[].inTreeModulesToRemove	Extra in-tree modules to unload for this mapping; moduleName is prepended automatically.
inTreeModulesToRemove (container level)	Auto-set to [moduleName] by the controller — the in-tree module is always unloaded before OoT insertion.
modulesLoadingOrder	Softdep-style loading order for multi-module drivers (first element must be moduleName, >=2 entries). Passed through as-is.
firmwarePath	In-container path for firmware files (maps to KMM ModprobeSpec.FirmwarePath).
registryTLS	Registry TLS options (insecure, insecureSkipTLSVerify) for the OoT image registry. Settable at top level and per mapping.

What's included

• KernelModuleSpec / KernelMappingSpec / RegistryTLSSpec CRD types with DeepCopy
• KMMReconciler sub-controller with CreateOrPatch, owner references, and deletion guard
• KMM ready-label gating for downstream DP/DRA/XPU DaemonSets
• Webhook validation and moduleName defaulting for kernelModule
• OpenShift module-loader SCC, ServiceAccount, and RBAC via Helm templates
• KMM Module RBAC (modules verbs) in the operator ClusterRole
• Controller tests (~1,070 lines) and webhook validation tests (~635 lines) via envtest
• Policy chart values and template for kernelModule configuration

Test results

Latest end-to-end run on OCP 4.22.9 SNO, 2× Intel BMG-G31 (Battlemage, PCI 8086:e223), kernel 5.14.0-687.35.1.el9_8.x86_64, KMM v2.7.0 + NFD pre-installed:

Scenario	Result
OoT + DRA (BMG-G31, in-tree driver replaced)	Module CR created with correct spec; KMM loaded OoT xe (kmmStatus: 1/1); ready label gated DP/DRA/XPU until load; DRA driver + XPU Manager 1/1; ResourceSlice populated with both GPUs; DRA isolation test passed (/dev/dri/ present in claimed container, absent otherwise); status errors cleared once healthy
CRD schema (simplified API)	New fields present, dropped fields absent; moduleName defaults to xe; regexp required (webhook-enforced)

Corroborating results from earlier runs (OCP 4.22.2, Arc Pro B70 with SR-IOV VFs):

Scenario	Result
In-tree, no KMM	DRA resources created, no Module CR, GPU workload OK, kmmStatus: N/A, clean deletion
OoT + DP	Module CR created, gpu.intel.com/xe=8 registered, GPU workload OK

Known limitations

xe can't unload with active VFs. modprobe -rv cannot unload xe while SR-IOV VFs are active. KMM is adding modprobe.d support (kubernetes-sigs/kernel-module-management#1324) that could work around this in the future.

This PR was written in part with the assistance of generative AI.

[tkatila]

Some comments related to the CRD changes. I think the Module CR is somewhat complex as it has same fields in different places and would like to not copy them as they are. If possible.

[tkatila]

Thanks @abyrne55 for the changes! I'll try to use this on my end in the next couple of days.

[tkatila]

Some notes from testing the in-cluster build.

Couple of things that I noticed:

• In Ubuntu, the worker image fails to set the firmware load path. I "workarounded" that by building a custom worker image where the user is 0 (not 201).
  • I think I tried setting the worker config to use user 0, but that didn't help.
• In Ubuntu26.04, the kernel prevents unloading the in-tree xe as there are devices using it.
  • By blacklisting the xe driver, the initial loading works, but the problem surfaces again when trying to remove the ClusterPolicy (=removing the Module thus unloading the oot kmd).
  • I think this is noted here: https://github.com/kubernetes-sigs/kernel-module-management/blob/main/docs/enhancements/0005-modprobed-config.md

[tkatila]

For fun, tested how KMMO handles update from x->y version. Updated ClusterPolicy with a different versions, and KMMO built the new kmd container and updated the driver. Nice 👍

Though, this was with a slightly hacked KMMO container to allow unbinding in the worker container.

[abyrne55]

Thank you for the thorough testing, @tkatila! See my replies above w.r.t. your inline comments. As for the remaining items:

In Ubuntu, the worker image fails to set the firmware load path. I "workarounded" that by building a custom worker image where the user is 0 (not 201). I think I tried setting the worker config to use user 0, but that didn't help.

Sounds like a potential CRI-O vs containerd difference in how privileged non-root containers get capabilities. I'll see if I can reproduce within OCP tomorrow and file an upstream bug as needed.

In Ubuntu26.04, the kernel prevents unloading the in-tree xe as there are devices using it.
By blacklisting the xe driver, the initial loading works, but the problem surfaces again when trying to remove the ClusterPolicy (=removing the Module thus unloading the oot kmd). I think this is noted here: https://github.com/kubernetes-sigs/kernel-module-management/blob/main/docs/enhancements/0005-modprobed-config.md

Indeed, this aligns with the RHEL/OCP behavior I mention at the bottom of the PR description. You're spot-on that the modprobed enhancement would allow us to tackle that. I'll try to get some timeline insight there. In the meantime, just want to confirm: this is what you've been doing as an unbinding workaround, yes?

# Zero out VFs, if SR-IOV is active
echo 0 > /sys/bus/pci/devices/$GPU_PCI_ADDRESS/sriov_numvf

# Unbind PF(s)
echo $GPU_PCI_ADDRESS > /sys/bus/pci/drivers/xe/unbind

[tkatila]

Sounds like a potential CRI-O vs containerd difference in how privileged non-root containers get capabilities. I'll see if I can reproduce within OCP tomorrow and file an upstream bug as needed.

Thanks. I'll try to reproduce the issue again and see if using the worker config fixes it. But I think my past experience was that it didn't fix it.

Indeed, this aligns with the RHEL/OCP behavior I mention at the bottom of the PR description. You're spot-on that the modprobed enhancement would allow us to tackle that. I'll try to get some timeline insight there. In the meantime, just want to confirm: this is what you've been doing as an unbinding workaround, yes?

# Zero out VFs, if SR-IOV is active
echo 0 > /sys/bus/pci/devices/$GPU_PCI_ADDRESS/sriov_numvf

# Unbind PF(s)
echo $GPU_PCI_ADDRESS > /sys/bus/pci/drivers/xe/unbind

Yes, that's the gist of it. Or iterate over all the BDFs under the xe driver and unbind them.

[tkatila]

fyi, if you add this, the "Build all" check should start passing:

diff --git a/build/operator/Dockerfile b/build/operator/Dockerfile
index 0a09c39..f288eb4 100644
--- a/build/operator/Dockerfile
+++ b/build/operator/Dockerfile
@@ -53,8 +53,10 @@ RUN apt-get update -y && apt-get --no-install-recommends -y install file && \
     chmod +x checksec && \
     ./checksec --file=/workspace/manager --output=csv | grep -q "$CHECKSEC_REF"
 
-RUN go get github.com/google/go-licenses && \
-    go run github.com/google/go-licenses save ./cmd/ --save_path licenses && \
+# GOROOT is passed explicitly so go-licenses can recognize stdlib packages even
+# when `go list` re-execs a different toolchain than the one that built it.
+RUN go install github.com/google/go-licenses@v1.6.0 && \
+    GOROOT="$(go env GOROOT)" go-licenses save ./cmd/ --save_path licenses && \
     cp /workspace/LICENSE /workspace/licenses/LICENSE.intel-gpu-base-operator
 
 ARG BASE_IMAGE

[abyrne55]

Update on the firmwarePath bug: I wasn't able to reproduce this on OCP, but I was able to reproduce it on Ubuntu+k3s, which points at a containerd-vs-CRI-O capability difference plus an image-uid difference. I've filed kubernetes-sigs/kernel-module-management#1337

Root cause for those curious: KMM seems to ignore the worker config UID whenever firmwarePath is set, so the container always runs as the worker image's baked-in USER. OCP's KMM worker image (registry.redhat.io/kmm/kernel-module-management-worker-rhel9:2.7.0) runs as root, not uid 201, so writing to the root-owned /sys/module/firmware_class/parameters/path just works. Even when I forced runAsUser: 201, CRI-O allows privileged containers like this one to keep full effective capabilities, so the sysfs write still succeeds. Vanilla k8s/containerd doesn't give the privileged-but-not-root worker container CAP_DAC_OVERRIDE in its effective set, leading to what @tkatila observed.

[tkatila]

I tested the latest content and it seems fine. 👍

If you @abyrne55 rebase this to main and maybe squash/cleanup the commits a bit, we can merge this.

After the merge, I think the overall KMM support needs a section to README (or maybe a page of its own). We need to mark down the current known limitations and limited support in the Xe OoT KMD.

[tkatila]

Something went wrong with the rebase. There are some reverts in the change set that shouldn't be there (codeql.yml, scorecard.yml, node-feature-rules-gpu.yaml etc.)

[abyrne55]

Good catch @tkatila. Should be all set now 🚀

[tkatila]

lgtm