Pace and count the mapper's writes, not just the sort's

[JamesKane]

The gap

The write pacing and byte accounting that came out of the 2026-08-13 WindowServer teardown shipped as a module inside navigator-analysis. That crate holds the revert, sort, markdup and CRAM stages — but not the mapper, which lives in navigator-align, a leaf crate that cannot depend on it and should not.

So half the pipeline was covered. Unpaced and uncounted:

Writer	Size on a 30x WGS
mapped.bam	~60 GB
minimizer index (.mmi)	8.93 GB
final CRAM	tens of GB

The phase-5 run log reads 0 MB/s straight through the longest stage in the job for exactly that reason — not a quiet stage, an unmeasured one. And pacing the sort while the mapper wrote unpaced left the original failure mode reachable, because the resource notice macOS filed was against the process, not against a stage. Fixing half the writers was closer to no fix than it looked.

The change

navigator-resource, a new leaf crate holding PacedFile, the process-wide byte counter, and ResourceWatch, depended on by both halves of the pipeline. It has to be a crate rather than a module: a counter only means something if there is exactly one of it. navigator-app takes it directly rather than reaching through navigator-analysis, since neither half is the owner. The move itself is a git mv — 89% similarity, doc changes only, no logic.

Both CRAM encoders were taking build_from_path, which opens the file themselves; build_from_writer makes the writer ours to wrap.

Two things from reading the same write path

• Buffers. Every one of those writers sat behind BufWriter's 8 KB default while the encoders above hand down 64 KB BGZF blocks and multi-MB CRAM containers, so the buffer was coalescing nothing. Now 1 MB, matching bamio.
• Sync at the end of each stage output. This matters most for mapped.bam: its BGZF end-of-file block is precisely what a resumed run reads to decide the file can be trusted, and trusting a marker that was still only a page-cache promise is how 59 GB was destroyed during phase 5. Also the .mmi, whose atomic rename otherwise publishes contents the disk has not acknowledged.

Test

navigator-align carries the regression test — write a BAM through AlignmentWriter, assert the shared counter moved. Greater-than rather than an exact figure, since the counter is shared with anything else in the test binary; the claim under test is that these bytes are counted at all. It would have failed before this change.

984 pass, 0 fail (983 + the new one). Clippy clean under -D warnings, fmt clean.

Deliberately not done

• finalize.rs's .bai is left unpaced — a few MB, where the syscalls and log noise would buy nothing.
• Writers outside the realignment pipeline were not audited.
• NAVIGATOR_SORT_MB still defaults to 512 MB (688 spill runs on a 128 GB machine).

🤖 Generated with Claude Code