fix: data race in the /metrics handler - #1406
Open
nisay759 wants to merge 1 commit into
Open
Conversation
The /metrics handler assigned h.options.Stats (a *Metrics) to a local variable, which copied the pointer rather than the struct. Every request therefore mutated the single shared Metrics struct when setting Cache, Cpu, Memory and Network, racing with concurrent /metrics requests, and read the counters non-atomically while the protocol servers updated them with atomic adds. Snapshot the counters into a local value using atomic loads instead, so the shared struct is never written and the encoded values are consistent. Add a regression test asserting the handler leaves the shared struct untouched, plus a concurrent metrics test that fails under -race on the old code. Co-Authored-By: Claude <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix: data race in the
/metricshandlerSource:
fix/metrics-race-and-session-countTarget:
devCommit:
27faf5cDiffstat: 2 files changed, 104 insertions(+), 2 deletions(-)
Summary
metricsHandlerassignsh.options.Stats— a*Metricsshared by every protocol server — to a local variable. That copies the pointer, not the struct, so each request writes into the one shared struct and reads its counters non-atomically while the protocol servers update them with atomic adds.The fix snapshots the counters into a local value using atomic loads, so the shared struct is never written and the encoded numbers are self-consistent.
Scope is deliberately narrow: only the handler's read path changes. The session counters themselves are untouched — the snapshot simply reads
SessionsandSessionsTotalalongside the protocol counters.The bug
Options.Statsis a*Metrics(pkg/server/server.go:133). The handler began with:Because
interactMetricsis the same pointer, those four assignments mutate the shared struct.Two distinct races follow:
/metricsrequests assign.Cache,.Cpu,.Memoryand.Networkon the same struct.atomic.AddUint64/atomic.AddInt64(dns_server.go,http_server.go,smtp_server.go,ftp_server.go,smb_server.go, and eleven call sites inldap_server.go).The fix
Every writer was confirmed to use atomic adds, so the loads pair correctly.
Impact
Confined to observability — nothing gates on these counters, and the interaction-capture pipeline is untouched. But
/metricsis exactly the endpoint an operator scrapes on an interval, so the racy path is the one under sustained concurrent load in production, andSessionsis the only in-app signal of how many clients a server is carrying.Test plan
New file
pkg/server/metrics_race_test.go(88 lines, 2 tests):TestMetricsHandlerDoesNotMutateSharedStats— asserts the handler leavesoptions.Stats.Cache/Cpu/Memory/Networknil. A deterministic check that fails without-race.TestMetricsHandlerConcurrent— four goroutines updating counters against four goroutines hittingmetricsHandler. Intended for-race.Both were checked against the unfixed handler, confirming they fail for the right reasons rather than passing vacuously:
TestMetricsHandlerDoesNotMutateSharedStatsExpected nil, but got: &storage.CacheMetrics{...}TestMetricsHandlerConcurrentrace detected during execution of testWith the fix applied:
go.mod/go.sumunchanged.Reviewer notes
CI would not have caught this.
.github/workflows/build-test.yml:33has a step named "Race Condition Tests" that runs:That only compiles with the detector enabled — no test is ever executed under
-race. Changing it togo test -race ./...would make the step do what its name claims. Left out of this PR as unrelated, but recommended as a follow-up.Misc note
The code was produced using Claude AI