COO-1597: Raise memory limit of observability-operator deploy by muellerfabi · Pull Request #994 · rhobs/observability-operator

muellerfabi · 2026-02-10T12:37:01Z

On a larger OCP 4.18 cluster with 66 nodes observability-operator 1.3.0 gets OOMKilled right after start.

Issue is reported in https://access.redhat.com/support/cases/#/case/04368491
Issue is tracked in https://issues.redhat.com/browse/COO-1597

Update due to missing information about the actual issue:
We are aware of the fact that it is possible to set resource requests and limits in the subscription. We use the following as a workaround:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-observability-operator
  namespace: openshift-operators-redhat
spec:
  channel: stable
  config:
    resources:
      limits:
        memory: 3Gi
     requests:
       memory: 500Mi

The problem is that the given config is set on all COO components equally:

obo-prometheus-operator
obo-prometheus-operator-admission-webhook
observability-operator
perses-operator

Because of the elevated memory usage of perses in contrast to the other components, we were forced to set the memory limit to 3Gi.
Now all the components have a needlessly too high memory limit:

$ oc get deploy -o custom-columns=NAME:.metadata.name,RESOURCES:.spec.template.spec.containers[0].resources
NAME                                        RESOURCES
logging                                     map[]
loki-operator-controller-manager            map[]
obo-prometheus-operator                     map[limits:map[memory:3Gi] requests:map[memory:500Mi]]
obo-prometheus-operator-admission-webhook   map[limits:map[memory:3Gi] requests:map[memory:500Mi]]
observability-operator                      map[limits:map[memory:3Gi] requests:map[memory:500Mi]]
perses-operator                             map[limits:map[memory:3Gi] requests:map[memory:500Mi]]


$ oc adm top po
NAME                                                        CPU(cores)   MEMORY(bytes)   
logging-7c8b5bfdf4-v5p67                                    1m           23Mi            
loki-operator-controller-manager-7c5b4ffbfb-wvt9b           49m          5622Mi          
obo-prometheus-operator-545cdc864f-wxmzh                    61m          317Mi           
obo-prometheus-operator-admission-webhook-57d54bf6d-4p6j6   1m           11Mi            
obo-prometheus-operator-admission-webhook-57d54bf6d-x5pfh   1m           12Mi            
observability-operator-595c984dfb-24lsc                     3m           536Mi           
perses-operator-5fc9687477-8g9jc                            11m          1701Mi

openshift-ci · 2026-02-10T12:37:07Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: muellerfabi
Once this PR has been reviewed and has the lgtm label, please assign simonpasquier for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2026-02-10T12:37:13Z

Hi @muellerfabi. Thanks for your PR.

I'm waiting for a rhobs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

simonpasquier · 2026-02-16T13:49:58Z

/retitle COO-1597: Raise memory limit of observability-operator deploy

while it might help the reported case, I think that we should also document how users can customize the out-of-the box limits and where the current limits fit because I'm sure that we'll get other reports in the future (https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/subscription-config.md#resources).

openshift-ci-robot · 2026-02-16T13:50:04Z

@muellerfabi: This pull request references COO-1597 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.22.0" version, but no target version was set.

Details

In response to this:

On a larger OCP 4.18 cluster with 66 nodes observability-operator 1.3.0 gets OOMKilled right after start.

Issue is reported in https://access.redhat.com/support/cases/#/case/04368491
Issue is tracked in https://issues.redhat.com/browse/COO-1597

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

jan--f · 2026-04-24T11:33:55Z

@jgbernalp wdyt? It's a small adjustment, so not sure how many user this will help out of the box.

jgbernalp · 2026-04-24T16:09:13Z

@jgbernalp wdyt? It's a small adjustment, so not sure how many user this will help out of the box.

Is there a way for users to configure these limits? this might apply for this case. @muellerfabi do you have a pprof from this cluster that we can analyze?

simonpasquier · 2026-04-27T08:06:51Z

@jgbernalp yes it's possible (https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/subscription-config.md#resources) but the original report complained that it "should" just work out of the box.

duritong · 2026-04-27T12:58:50Z

@simonpasquier the main issue with: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/subscription-config.md#resources is that it overwrites the resource constraints globally and thus for example if you have a deployment with the actual operator and for example kube-rbac-proxy sidecar (very typical), your large requirements will also apply there, see redhat-cop/patch-operator#76 as an example from another operator.

openshift-ci-robot · 2026-06-19T10:22:27Z

@muellerfabi: This pull request references COO-1597 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "5.0.0" version, but no target version was set.

Details

In response to this:

On a larger OCP 4.18 cluster with 66 nodes observability-operator 1.3.0 gets OOMKilled right after start.

Issue is reported in https://access.redhat.com/support/cases/#/case/04368491
Issue is tracked in https://issues.redhat.com/browse/COO-1597

Update due to missing information about the actual issue:
We are aware of the fact that it is possible to set resource requests and limits in the subscription. We use the following as a workaround:
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: cluster-observability-operator
 namespace: openshift-operators-redhat
spec:
 channel: stable
 config:
   resources:
     limits:
       memory: 3Gi
    requests:
      memory: 500Mi
The problem is that the given config is set on all COO components equally:

obo-prometheus-operator

obo-prometheus-operator-admission-webhook

observability-operator

perses-operator

Because of the elevated memory usage of perses in contrast to the other components, we were forced to set the memory limit to 3Gi.
Now all the components have a needlessly too high memory limit:
$ oc get deploy -o custom-columns=NAME:.metadata.name,RESOURCES:.spec.template.spec.containers[0].resources
NAME                                        RESOURCES
logging                                     map[]
loki-operator-controller-manager            map[]
obo-prometheus-operator                     map[limits:map[memory:3Gi] requests:map[memory:500Mi]]
obo-prometheus-operator-admission-webhook   map[limits:map[memory:3Gi] requests:map[memory:500Mi]]
observability-operator                      map[limits:map[memory:3Gi] requests:map[memory:500Mi]]
perses-operator                             map[limits:map[memory:3Gi] requests:map[memory:500Mi]]


$ oc adm top po
NAME                                                        CPU(cores)   MEMORY(bytes)   
logging-7c8b5bfdf4-v5p67                                    1m           23Mi            
loki-operator-controller-manager-7c5b4ffbfb-wvt9b           49m          5622Mi          
obo-prometheus-operator-545cdc864f-wxmzh                    61m          317Mi           
obo-prometheus-operator-admission-webhook-57d54bf6d-4p6j6   1m           11Mi            
obo-prometheus-operator-admission-webhook-57d54bf6d-x5pfh   1m           12Mi            
observability-operator-595c984dfb-24lsc                     3m           536Mi           
perses-operator-5fc9687477-8g9jc                            11m          1701Mi          

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

muellerfabi · 2026-06-19T10:35:25Z

I updated the initial comment with further information, why overwriting resources in the Subscription kind is not ideal.

@jgbernalp Is there a way for users to configure these limits? this might apply for this case. @muellerfabi do you have a pprof from this cluster that we can analyze?
A must-gather from the affected cluster could be uploaded to the support case, if that helps?

Raising the memory limit to 550Mi is sufficient until the cluster grows to 90 nodes or so.
Maybe it was better to drop the limit entirely. Cluster-Operators usually do not have limits.
WDYT?

simonpasquier · 2026-06-19T12:41:14Z

Raising the memory limit to 550Mi is sufficient until the cluster grows to 90 nodes or so.

I'm worried that while it works for this cluster, we'll hear about other clusters still hitting the limit.

Maybe it was better to drop the limit entirely. Cluster-Operators usually do not have limits.
WDYT?

My initial assumption was that resource limits were a strong requirement but apparently not: https://sdk.operatorframework.io/docs/best-practices/managing-resources/#general-guidelines
We'd need to discuss the downsides of removing the limits but it might be the best course of action.

coderabbitai · 2026-07-07T09:20:40Z

Warning

Review limit reached

@muellerfabi, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 54 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: de1ba56c-516f-4ae1-bac0-9d78cbfff297

📥 Commits

Reviewing files that changed from the base of the PR and between cc2c8c2 and 2d9f94d.

📒 Files selected for processing (5)

bundle/manifests/observability-operator.clusterserviceversion.yaml
deploy/dependencies/kustomization.yaml
deploy/olm/syncselector-template.yaml
deploy/operator/observability-operator-deployment.yaml
deploy/perses/perses-operator-deployment.yaml

📝 Walkthrough

Walkthrough

This change removes explicit CPU/memory resource limits across multiple deployment manifests, replacing resources.limits with an empty object (limits: {}) while preserving existing resources.requests values. Affected resources include the ClusterServiceVersion bundle manifest (four containers), the observability-operator deployment, the perses-operator deployment, and the OLM SelectorSyncSet template. The template also removes the RESOURCE_LIMIT_CPU and RESOURCE_LIMIT_MEMORY parameters and adds a new required RESOURCE_REQUEST_CPU parameter with a default of 100m.

Estimated code review effort: 2 (Simple) | ~10 minutes

Changes

File(s)	Change Summary
`bundle/manifests/observability-operator.clusterserviceversion.yaml`	Removed CPU/memory limits (set to `{}`) for four containers
`deploy/operator/observability-operator-deployment.yaml`	Removed CPU/memory limits (set to `{}`) for the operator container
`deploy/perses/perses-operator-deployment.yaml`	Removed CPU/memory limits (set to `{}`) for the perses-operator container
`deploy/olm/syncselector-template.yaml`	Removed `RESOURCE_LIMIT_CPU`/`RESOURCE_LIMIT_MEMORY` params, added `RESOURCE_REQUEST_CPU` param; set limits to `{}` in three subscription resource blocks

Sequence Diagram(s)

Not applicable — this change consists of configuration edits without new control flow or interaction logic.

Related issues: None specified in the provided information.

Related PRs: None specified in the provided information.

Suggested labels: deployment, configuration

Suggested reviewers: None specified in the provided information.

🐰 No more limits capping what we can do,
Requests remain, but ceilings are through,
Four containers freed from their bounded fate,
Templates updated, parameters trade,
A hop toward flexibility, clean and true.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The title is misleading: the PR removes resource limits from several deployments, not just raises the observability-operator memory limit.	Rename it to reflect the actual change, e.g. 'Remove resource limits from COO deployments'.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description directly explains the OOM issue and the resource-limit workaround this PR changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

🧹 Nitpick comments (1)

bundle/manifests/observability-operator.clusterserviceversion.yaml (1)
956-956: 🧹 Nitpick | 🔵 Trivial

Removing limits drops OOM guardrails for all four COO containers.

With limits: {}, none of obo-prometheus-operator, its admission webhook, observability-operator, or perses-operator have a memory ceiling — a leak/regression in any of them can now consume unbounded node memory instead of being OOMKilled in isolation. This matches the PR's stated direction (per discussion, limits may be more harmful than helpful here), but worth confirming this tradeoff is accepted for all four components, not just perses-operator which triggered COO-1597.

Also applies to: 1027-1027, 1094-1094, 1166-1166
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@bundle/manifests/observability-operator.clusterserviceversion.yaml` at line
956, The CSV manifest is removing memory limits from all four COO containers,
not just perses-operator, so confirm this is an intentional product decision
across obo-prometheus-operator, its admission webhook, observability-operator,
and perses-operator. Review the container specs in
observability-operator.clusterserviceversion.yaml and either restore appropriate
limits where guardrails are still desired or keep limits removed only if the
no-limits tradeoff is explicitly accepted for every container.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@bundle/manifests/observability-operator.clusterserviceversion.yaml`:
- Line 956: The CSV manifest is removing memory limits from all four COO
containers, not just perses-operator, so confirm this is an intentional product
decision across obo-prometheus-operator, its admission webhook,
observability-operator, and perses-operator. Review the container specs in
observability-operator.clusterserviceversion.yaml and either restore appropriate
limits where guardrails are still desired or keep limits removed only if the
no-limits tradeoff is explicitly accepted for every container.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 35b05cf3-0ed4-4c95-8295-db4945b3f278

📥 Commits

Reviewing files that changed from the base of the PR and between 8ee6ecc and cc2c8c2.

📒 Files selected for processing (4)

bundle/manifests/observability-operator.clusterserviceversion.yaml
deploy/olm/syncselector-template.yaml
deploy/operator/observability-operator-deployment.yaml
deploy/perses/perses-operator-deployment.yaml

muellerfabi · 2026-07-07T09:24:17Z

I updated my commit deleting resource limits for the components.

KCS article https://access.redhat.com/solutions/6958023 states, beside some rationale:

Therefore, cluster components SHOULD NOT be configured with resource limits.

However, cluster components MUST declare resource requests for both CPU and memory.

danielmellado · 2026-07-15T13:03:17Z

/ok-to-test

openshift-ci Bot requested review from danielmellado and lihongyan1 February 10, 2026 12:37

openshift-ci Bot added the needs-ok-to-test label Feb 10, 2026

openshift-ci Bot changed the title ~~COO-1597 Raise memory limit of observability-operator deploy~~ COO-1597: Raise memory limit of observability-operator deploy Feb 16, 2026

openshift-ci-robot added the jira/valid-reference label Feb 16, 2026

muellerfabi force-pushed the coo-1597 branch from 15ea476 to cc2c8c2 Compare July 7, 2026 09:20

coderabbitai Bot reviewed Jul 7, 2026

View reviewed changes

remove resource limits

2d9f94d

muellerfabi force-pushed the coo-1597 branch from cc2c8c2 to 2d9f94d Compare July 7, 2026 09:25

openshift-ci Bot added ok-to-test and removed needs-ok-to-test labels Jul 15, 2026

Uh oh!

Conversation

muellerfabi commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented Feb 10, 2026

Uh oh!

openshift-ci Bot commented Feb 10, 2026

Uh oh!

simonpasquier commented Feb 16, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Feb 16, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jan--f commented Apr 24, 2026

Uh oh!

jgbernalp commented Apr 24, 2026

Uh oh!

simonpasquier commented Apr 27, 2026

Uh oh!

duritong commented Apr 27, 2026

Uh oh!

openshift-ci-robot commented Jun 19, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

muellerfabi commented Jun 19, 2026

Uh oh!

simonpasquier commented Jun 19, 2026

Uh oh!

coderabbitai Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

muellerfabi commented Jul 7, 2026

Uh oh!

danielmellado commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

muellerfabi commented Feb 10, 2026 •

edited

Loading

simonpasquier commented Feb 16, 2026 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Feb 16, 2026 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Jun 19, 2026 •

edited by openshift-ci Bot

Loading

coderabbitai Bot commented Jul 7, 2026 •

edited

Loading