[release-4.22] OCPBUGS-87022: fix concurrent map race in project authorization cache - #653
Conversation
addSubjectsToNamespace and deleteNamespaceFromSubjects mutate subjectRecord.namespaces (a sets.String / map) in place while List() iterates the same map from HTTP request goroutines. This causes a fatal "concurrent map iteration and map write" panic that crashes openshift-apiserver pods intermittently. Use a dual-path strategy: during full cache invalidation, where new stores are private to the writer goroutine, mutate in place for zero-copy performance. During incremental updates, where stores are shared with concurrent readers, use copy-on-write to create new subjectRecord objects with copied namespaces sets. Group all three cache stores behind an atomic.Pointer so they swap as a single unit during full invalidation, ensuring readers see a consistent view. This avoids the lock contention that caused the previous mutex fix (PR openshift#267) to be reverted (PR openshift#326), while also avoiding the O(n²) allocation overhead of unconditional copy-on-write.
Benchmark synchronize() at varying namespace × user scales (10/10 through 1000/1000) with the cache always expired, forcing full invalidation on every iteration.
Benchmark the incremental COW path with varying levels of subject duplication (D=1 for no duplicates, D=10 for 10x duplicates per user). This exercises the scenario where broken upstream dedup in AllowedSubjects causes redundant COW copies in addSubjectsToNamespace during incremental cache updates.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
@openshift-cherrypick-robot: Jira Issue OCPBUGS-84534 has been cloned as Jira Issue OCPBUGS-87022. Will retitle bug to link to clone. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
Bot
changed the title
openshift-ci-robot
added
jira/severity-critical
|
@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-87022, which is valid. The bug has been moved to the POST state. 7 validation(s) were run on this bug
The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@openshift-cherrypick-robot: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/verified by "TestAuthorizationCacheRace" |
|
/lgtm |
|
/approve |
|
@sanchezl: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
verified
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: sanchezl The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/label backport-risk-assessed |
openshift-ci
Bot
added
the
backport-risk-assessed
7fa546d
into
openshift:release-4.22
|
@openshift-cherrypick-robot: Jira Issue Verification Checks: Jira Issue OCPBUGS-87022 Jira Issue OCPBUGS-87022 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@openshift-cherrypick-robot: new pull request created: #660 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Fix included in release 4.22.0-0.nightly-2026-06-11-035725 |
7 participants
This is an automated cherry-pick of #642
/assign sanchezl
/cherrypick release-4.21 release-4.20 release-4.19 release-4.18