Dependency discussion: rapidgzip-rust for parallel gzip input (pure Rust)

[BenjaminDEMAILLE]

Raising this before any PR, per the dependency rule in CONTRIBUTING.md. It is a pure-Rust crate, so it does not carry the objections that closed #162 and #177, but it is still a dependency and the decision is yours.

What it is

Crate	rapidgzip-rust, @rob-p COMBINE-lab (the salmon / alevin-fry group)
Licence	BSD-3-Clause / MIT / Apache-2.0 triple
Non-Rust dependencies	none — no cc, no cmake, no system library
What it does	Parallel gzip decompression, treating a gzip file as independently verifiable members; handles concatenated gzip and BGZF
Interfaces	a push API to std::io::Write, and a std::io::Read + Send adapter that drops into an existing parser

Why it might matter here

Sequencing FASTQ is gzipped. Effectively always. Every ENA and SRA download, every sequencing facility delivery, every 10x fastqs/ directory. This repository assumes it too: CONTRIBUTING.md tells a contributor to fetch ERR12389696_1.fastq.gz from ENA and keeps it compressed, and test/data_setup.sh, test/run_tests.sh and the rest of the harness read .fastq.gz throughout. Uncompressed FASTQ is the exception, not the rule.

So this is not about one mode. Decompression sits on the critical path of every rustar run that reads real data — bulk RNA-seq, two-pass, solo, chimeric, all of it.

Today src/io/fastq.rs detects .gz and hands it to flate2 with the zlib-rs backend. That is pure Rust and already faster than miniz_oxide, but it is single-threaded per file: one producer thread inflates while up to --runThreadN workers wait on it. The more cores a run is given, the larger the share of the machine that can be idle behind one decompressor.

I have not measured that share, and I do not want to open a PR on the assumption that it is large. What I can give is the shape: a human PBMC run today read 1.6 GB of gzip for 20 M read pairs and finished the whole alignment in about 2.5 minutes at 14 threads. If inflating that input is a meaningful part of the 2.5 minutes, this crate is the fix. If it is not, the dependency buys nothing and this should be closed.

What I would do before proposing anything

1. Profile a gzipped-input run and get the actual share of wall-clock spent inflating, at 1 / 8 / 16 threads. The interesting number is how that share grows with thread count, since that is where a serial producer hurts.
2. If it is large, A/B the two decompressors on identical input, with interleaved medians, and confirm byte-identical output.
3. Only then a PR, carrying those numbers.

The honest counterweights

• flate2 is already a dependency and already pure Rust. This would be a second decompressor, not a replacement for a C one, so the tree grows rather than trades.
• The crate is young. Its own README describes "a substantial generic-stream scaling gap against ISA-L-enabled C++ rapidgzip", so it is not yet at the ceiling of the approach.
• Its BGZF support overlaps noodles-bgzf, already here for BAM. Two paths to one format is a maintenance question by itself.
• Parallel decompression must not make output depend on thread count. Determinism is a correctness property here, so record ordering would need the same guarantee the aligner already has.

Happy to run step 1 and report the number either way — that is a measurement, not a PR, and it settles whether this is worth your time.

[BenjaminDEMAILLE]

Measured, and it says no

I said I would not propose this without a number, so here is the number. 10x pbmc_1k_v3, 20 M read pairs, GRCh38, one run per cell: the same input as gzip (1.6 GB) and decompressed on disk (6.7 GB), so the only difference is whether flate2 runs.

threads	gzipped input	plain input	difference
1	102.1 s	99.0 s	+3.1%
8	138.1 s	139.0 s	−0.6%
14	95.9 s	95.3 s	+0.6%

Decompression is not a bottleneck here. The hypothesis was that a single-threaded producer would serialise the pipeline and that the cost would grow with thread count. If that were happening, plain input would pull away at 14 threads. It does not: 95.9 against 95.3, which is noise.

What is wrong with this measurement, stated rather than buried

One round per cell, not interleaved. That is exactly the weakness my own A/B harness in #171 exists to prevent, and I did not use it here because the harness is not merged and this was meant to be a quick screen. The 8-thread row is visibly off — slower than both 1 and 14 threads on both inputs — which is a thermal or scheduling artefact and a fair warning about how much any single row is worth.

What the measurement can still carry is the absence of a large effect. A serial decompressor that mattered would not hide inside a ±3% spread at 14 threads; it would show as tens of percent. So the screen is good enough to rule out the thing that would have justified the dependency, even if it is not good enough to quantify a small one.

Recommendation

Close this. flate2 with zlib-rs is keeping up with 14 aligner threads on real gzipped 10x input, and adding a second decompressor to the tree buys nothing measurable. If a future change makes alignment much faster — the batched seed search in the roadmap's P3 is the candidate — the balance could shift, and that would be the moment to re-run this rather than now.

Closing unless you would rather I redo it properly with interleaved rounds first.

[BenjaminDEMAILLE]

Closing on the measurement above: gzipped and plain input are within noise at 1, 8 and 14 threads on real 10x data, so a parallel decompressor buys nothing measurable today. Worth revisiting if alignment itself gets much faster — the batched seed search (roadmap P3) is the change that would shift the balance.