Size the spill buffers from the machine, and count records honestly

[JamesKane]

Supersedes #53, which GitHub auto-closed when its base branch (#52's) was deleted on merge. Same commit, rebased onto main; nothing about the change itself moved.

The constant

Both external sorts were sized by a fixed number: 512 MB for the coordinate sort, 256 MB for the revert's collator, identical on a laptop and on a 128 GB workstation. A 30x WGS spilled 688 runs, all of which the merge opens at once. Bounded memory by design, and it works — it is just a lot of fan-in bought for no reason on a machine sitting idle.

One rule, both stages

navigator_resource::spill_budget(var):

Clause	Why
a quarter of installed RAM	Total, not free — a stage whose run count depends on open browser tabs cannot be reproduced from a bug report.
never below 512 MB	The sort's existing default. No machine does worse than today.
never above 8 GB	88 runs against 44 is nothing next to 688 against 88, while the stable sort's scratch (half the buffer again) and the record vector's doubling stay real.
never above half of what is free now	The stable clause assumes an idle machine. When it isn't, an extra spilled run is cheap and swapping the buffer is not.

Unknown memory reads as the floor rather than as zero — the rule classify already follows. NAVIGATOR_SORT_MB / NAVIGATOR_REVERT_SORT_MB still override, and the job logs the figure it chose.

The part that wasn't optional

Sizing from the machine is only safe if the tally is honest, and it wasn't. heap_bytes charged a record for its name, sequence, qualities and CIGAR — and not for its tag dictionary. Every minimap2 record carries ~12 tags at 40 bytes an entry, so a "512 MB" buffer held closer to a gigabyte. The collator's flat 64 bytes of overhead was smaller than its own record struct.

That margin is harmless inside a hand-picked constant and dangerous inside a fraction of installed RAM. Both estimators now count the record's own size (RecordBuf = 176 bytes, measured), the tag dictionary, and per-allocation overhead. A test pins the tag-entry figure so a noodles upgrade that grows Value fails there rather than silently halving the buffer's honesty.

Effect

On a 128 GB machine: the 8 GB ceiling, roughly 84 spilled runs against 688. On 16 GB: a 4 GB buffer. On 8 GB with 6 free: 2 GB.

994 pass, 0 fail (984 + 10 new). Clippy clean under -D warnings, fmt clean — and all three platforms passed on #53 with this same tree.

Not done

• Peak is still somewhat above the budget: the record vector's spare capacity and the stable sort's scratch are not charged, only bounded by the ceiling.
• No WGS run has exercised the new sizing yet — the arithmetic is tested, the wall-clock claim is not.

🤖 Generated with Claude Code