[release-4.21] OCPBUGS-87022: fix concurrent map race in project authorization cache - #660
Conversation
addSubjectsToNamespace and deleteNamespaceFromSubjects mutate subjectRecord.namespaces (a sets.String / map) in place while List() iterates the same map from HTTP request goroutines. This causes a fatal "concurrent map iteration and map write" panic that crashes openshift-apiserver pods intermittently. Use a dual-path strategy: during full cache invalidation, where new stores are private to the writer goroutine, mutate in place for zero-copy performance. During incremental updates, where stores are shared with concurrent readers, use copy-on-write to create new subjectRecord objects with copied namespaces sets. Group all three cache stores behind an atomic.Pointer so they swap as a single unit during full invalidation, ensuring readers see a consistent view. This avoids the lock contention that caused the previous mutex fix (PR openshift#267) to be reverted (PR openshift#326), while also avoiding the O(n²) allocation overhead of unconditional copy-on-write.
Benchmark synchronize() at varying namespace × user scales (10/10 through 1000/1000) with the cache always expired, forcing full invalidation on every iteration.
Benchmark the incremental COW path with varying levels of subject duplication (D=1 for no duplicates, D=10 for 10x duplicates per user). This exercises the scenario where broken upstream dedup in AllowedSubjects causes redundant COW copies in addSubjectsToNamespace during incremental cache updates.
|
@openshift-cherrypick-robot: An error was encountered cloning bug for cherrypick for bug OCPBUGS-87022 on the Jira server at https://redhat.atlassian.net. No known errors were detected, please see the full error message for details. Full error message.
request failed. Please analyze the request body for more details. Status code: 400: {"errorMessages":[],"errors":{"customfield_10980":"Field does not support update 'customfield_10980'","customfield_10978":"Field does not support update 'customfield_10978'","customfield_10979":"Field does not support update 'customfield_10979'"}}
Please contact an administrator to resolve this issue, then request a bug refresh with DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@openshift-cherrypick-robot: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
4 participants
This is an automated cherry-pick of #653
/assign openshift-cherrypick-robot
/cherrypick release-4.20 release-4.19 release-4.18