Skip to content

fix: --runThreadN 1 ran on every core, not on one - #166

Open
BenjaminDEMAILLE wants to merge 2 commits into
scverse:mainfrom
BenjaminDEMAILLE:bd/fix-runthreadn-one
Open

fix: --runThreadN 1 ran on every core, not on one#166
BenjaminDEMAILLE wants to merge 2 commits into
scverse:mainfrom
BenjaminDEMAILLE:bd/fix-runthreadn-one

Conversation

@BenjaminDEMAILLE

@BenjaminDEMAILLE BenjaminDEMAILLE commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

--runThreadN 1 ran on every logical core. One line, and it was hiding in plain sight.

What was wrong

The rayon global pool was configured only when --runThreadN was greater than 1:

if usize::from(params.run_thread_n) > 1 {
    let _ = rayon::ThreadPoolBuilder::new()
        .num_threads(params.run_thread_n.into())
        .build_global();
}

Skipping the build at 1 does not give one thread. It leaves rayon's default, which is one worker per logical core. So the single value a user is most likely to pass when they mean "use one core" was the one value that used all of them.

Measured on 200 000 real reads, 16-core machine:

wall CPU
before 2.44 s 1300%
after 26.23 s 100%

The old 2.44 s was not a fast single-threaded run. It was a sixteen-way run wearing the wrong flag, which is also why a scaling curve taken with it looked absurd: --runThreadN 2 was slower than --runThreadN 1, because 2 actually meant 2 and 1 meant 16.

Why it matters beyond the flag reading falsely

  • A scheduler or a container allotted one CPU still spawns one worker per core.
  • On a shared machine the run oversubscribes every other job on it, silently.
  • With a thread-caching allocator each of those threads keeps its own heap. That is the precise cost the comment directly above this code says the pool sizing exists to avoid, so the guard defeated the stated purpose of the block it was guarding.
  • The thread-invariance checks were weaker than they read. A --runThreadN 1 leg that runs sixteen-wide is not a one-thread leg, so comparing it against --runThreadN 16 compared sixteen against sixteen.

Verification

Output-neutral, checked two ways on 200 000 real reads:

  • the new binary at --runThreadN 1 is byte-identical to the old binary at --runThreadN 1,
  • and with the fix in place, --runThreadN 1 and --runThreadN 8 produce identical records. The only differing byte is in the @PG CL: line, which records the command line and therefore the thread count itself.

That second check is the one the guard was preventing anyone from running. It passes.

With --runThreadN 1 now meaning one thread, the real scaling on this machine is 26.2 s at 1 thread against 2.42 s at 16, about 10.8×.

Gate: 560 lib + 26 integration tests, cargo clippy --all-targets -- -D warnings, cargo fmt --check, all green. No new dependency, no flag change, no output change.

Related

Found while taking a thread-scaling curve for #167, which is why the numbers there are trustworthy only after this: with the bug in place --runThreadN 2 measured slower than --runThreadN 1, because 2 meant 2 and 1 meant sixteen. #168 carries the allocation measurements from the same session.

The rayon global pool was configured only when `--runThreadN` was
greater than 1. Skipping the build at 1 does not give one thread: it
leaves rayon's default, which is one worker per logical core. So
`--runThreadN 1` ran the whole machine.

Measured on 200k reads, before: 2.44 s wall at **1300% CPU**. After:
26.23 s wall at 100% CPU. The old figure was not a fast single-threaded
run, it was a sixteen-way run wearing the wrong flag.

This matters beyond the flag reading falsely. A scheduler or a container
given one CPU gets sixteen worker threads; on a shared machine the run
oversubscribes every other job; and with a thread-caching allocator each
of those threads keeps its own heap, which is the very cost the comment
above this code says the pool sizing exists to avoid.

It also means the project's thread-invariance checks were weaker than
they read: the `--runThreadN 1` leg was not a one-thread leg. Verified
now that it is one: records are byte-identical between 1 and 8 threads
on 200k real reads, and byte-identical to the previous binary's output
at `--runThreadN 1`. Only the `@PG` `CL:` line differs between thread
counts, because it records the command line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant