Bug: SharedTrust and MLflow controllers block all reconciliation when RBAC is missing

[akram]

Summary

When cert-manager and/or OpenDataHub CRDs are installed on the cluster but the operator's ServiceAccount lacks RBAC to list/watch their resources, the SharedTrust and MLflow operand controllers register cache watchers that fail indefinitely. Because controller-runtime blocks all controllers until every cache is synced, this prevents the entire operator from reconciling — including unrelated controllers like clientregistration, agentruntime, and the authbridge webhook.

Symptoms

• clientregistration controller starts EventSources but never starts workers — no Keycloak client registration happens, no kagenti-keycloak-client-credentials-* secrets are created
• Pods stuck in ContainerCreating waiting for credential secrets that never appear
• Repeated cache.UnhandledError / Failed to watch log entries for:
  • *v1.Certificate (cert-manager): certificates.cert-manager.io is forbidden
  • *mlflow.DataScienceCluster (OpenDataHub): datascienceclusters.datasciencecluster.opendatahub.io is forbidden

Root cause

Both controllers use CRDExists() to detect whether their CRDs are installed, and if so, register watchers:

• c883d07 — SharedTrust controller watches cert-manager.io/Certificate
• 8f9cfa2 — MLflow operand controller watches datasciencecluster.opendatahub.io/DataScienceCluster

The CRDs exist on the cluster (cert-manager and OpenDataHub are installed), so the controllers register. But the Helm chart's ClusterRole does not include the necessary RBAC rules. The cache informers fail with forbidden, and controller-runtime waits indefinitely for cache sync before starting any controller's workers.

Environment

• ROSA HCP (OpenShift 4.x)
• cert-manager installed
• OpenDataHub installed (RHOAI)
• Operator installed via Helm chart

Affected commits

The regression appeared between 2073d03 (PR #382, working) and b93c5cf (PR #408, broken). The operator binary, webhook, and client registration code are unchanged — only the manager startup is blocked.

Suggested fix

Either:

1. Add the missing RBAC rules to the Helm chart's ClusterRole for cert-manager Certificates and DataScienceClusters
2. Make the controllers resilient — use a separate non-blocking cache or start these controllers in a goroutine that doesn't block the manager's WaitForCacheSync
3. Check RBAC before registering — extend CRDExists() to also verify the SA has list/watch permissions via a SubjectAccessReview before registering the watcher

Option 2 or 3 is preferred — optional controllers should not take down the entire operator when their RBAC is missing.

Workaround

Grant the missing permissions manually:

# cert-manager
oc create clusterrole kagenti-cert-manager-reader --verb=get,list,watch --resource=certificates.cert-manager.io
oc create clusterrolebinding kagenti-cert-manager-reader --clusterrole=kagenti-cert-manager-reader --serviceaccount=kagenti-system:controller-manager

# OpenDataHub
oc create clusterrole kagenti-datasciencecluster-reader --verb=get,list,watch --resource=datascienceclusters.datasciencecluster.opendatahub.io
oc create clusterrolebinding kagenti-datasciencecluster-reader --clusterrole=kagenti-datasciencecluster-reader --serviceaccount=kagenti-system:controller-manager

[r3v5]

I feel like one day I did have this issue with RBAC

[r3v5]

@akram Heads up — this is related to #426 (ServiceAccounts missing list/watch verbs). Same category of bug: Helm chart ClusterRole missing RBAC verbs that controller-runtime informers need. #426 is narrower (webhook hangs), this one is broader (blocks all controllers). Worth coordinating fixes so the ClusterRole gets a comprehensive RBAC audit rather than piecemeal patches.