fix: coalesce concurrent per-AZ placement group creation in mpicohort

[scttfrdmn]

Summary

• Actuator.ensurePlacementGroup (pkg/mpicohort/adapter.go) released its mutex between checking pgCreated[az] and calling CreatePlacementGroup, a classic check-then-act race. A cohort round launches N members concurrently, so multiple goroutines commonly race into this function for the same AZ at once — before the fix, more than one could observe "not created yet" and each call CreatePlacementGroup, defeating the once-per-AZ design that exists specifically to avoid the ~30s availability poll for every member of a round.
• Surfaced as a CI-only flake in TestActuator_PerAZPlacementGroup (AZ-b PG created 2 times, want 1) — confirmed not reproducible locally with -count=5 prior to this fix, consistent with a genuine race exposed only by CI's heavier scheduling/load, per the issue.
• Fixed by coalescing concurrent same-AZ callers onto a single in-flight CreatePlacementGroup call via a per-AZ done-channel (pgOnce): the first caller in does the create, the rest wait on its result. Callers for different AZs are not serialized against each other (verified by a dedicated test with a generous timing bound).
• Went with the done-channel approach over golang.org/x/sync/singleflight (already a transitive dep) to keep the change self-contained to the existing pgMu-guarded state, rather than introducing a second synchronization primitive for one call site.

Test plan

• go build ./..., go vet ./pkg/mpicohort/...
• go test ./pkg/mpicohort/... -race -count=20 — all green, including the previously-flaky TestActuator_PerAZPlacementGroup
• New regression tests: TestEnsurePlacementGroup_Concurrent (50 concurrent callers for one AZ, with an artificial delay inside the fake CreatePlacementGroup to reliably widen the race window — a fast in-memory fake without it may not interleave even with the race present, which is exactly why this only flaked in CI) and TestEnsurePlacementGroup_ConcurrentDifferentAZsNotSerialized.
• Verified TestEnsurePlacementGroup_Concurrent fails reliably (50/50 CreatePlacementGroup calls went through) against the pre-fix code, and passes with the fix, under -race -count=20.
• golangci-lint run ./pkg/mpicohort/... — 0 issues
• CHANGELOG.md updated under [Unreleased]

Fixes #514

[codecov]

Codecov Report

❌ Patch coverage is 68.00000% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/mpicohort/adapter.go	68.00%	5 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!