perf(sync): raise the default fetch ceiling floor to 50 threads#65
Open
srnnkls wants to merge 3 commits into
Open
perf(sync): raise the default fetch ceiling floor to 50 threads#65srnnkls wants to merge 3 commits into
srnnkls wants to merge 3 commits into
Conversation
Fetch is network-wait-bound: the sweep shows 96% io-wait at jobs=1 and aggregate CPU flat up to jobs=50, while 20 tiny sources drop 1.54s -> 0.95s going from jobs=16 to jobs=50 on 8 cores. Parked threads cost memory, not CPU, so the default ceiling becomes max(2*cores, 50); --jobs still overrides.
macOS awk rejects an unparenthesized ternary in a printf argument list; compute the wait percentage in a separate statement.
There was a problem hiding this comment.
Pull request overview
This PR increases the default Rayon thread-pool size used by sync when --jobs is not provided, raising the pool cap to at least 50 threads to better overlap network waits during fetch-heavy sync workloads. It also fixes the benchmark sweep script’s awk reporter to be compatible with BSD awk (macOS).
Changes:
- Raise default pool cap from
2×corestomax(2×cores, 50)(still bounded byunitsand still overridden by--jobs). - Update/expand unit tests to cover the new cap-floor behavior.
- Fix
benches/fetch_sweep.shawklogic for BSD awk portability.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| src/sync/resolve.rs | Adjusts default sync Rayon thread count cap (adds a minimum cap of 50) and updates tests/docs accordingly. |
| benches/fetch_sweep.sh | Makes the benchmark output formatter compatible with BSD awk by avoiding the prior inline conditional expression. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When
--jobsis unset, the default rayon pool size becomesmin(units, max(2×cores, 50))instead ofmin(units, 2×cores).--jobsstill overrides. Also fixes thebenches/fetch_sweep.shreporter, which BSD awk (macOS) rejected outright.Why
benches/fetch_sweep.sh(re-run 2026-07-10, 8-core machine; consistent with the 2026-06 baseline in the async-fetch scope findings) shows fetch is network-wait-bound and the pool ceiling — not the box — is the limit for many small sources:Aggregate CPU stays flat as jobs rise (heavy fixture: 13.96s → 12.59s across jobs 8→32), so the wider ceiling costs parked-thread memory, not cycles. The floor of 50 captures the measured win; boxes where 2×cores already exceeds 50 keep their existing ceiling.
Considered and rejected
Separating the fetch pool from the compute pool: the same sweep shows the machine is never CPU-saturated during sync (~0.44 of 8 cores busy on 11 medium/large repos) and there is no oversubscription penalty, so a dedicated compute pool has nothing to relieve.
Verification
cargo test: 1349 lib + all integration suites greencargo clippy --all-targets -- -D warnings: cleanbenches/fetch_sweep.shruns end-to-end again (reporter fix) — numbers above are from this branch's harness