solo: CellRanger .h5 matrices, written without libhdf5 (answers #177, stacked on #179)

[BenjaminDEMAILLE]

Stacked on #179 → #178 → #176 → #175 → #174 → #173. Merge in that order.

Closes the last piece of #172's output side, and it is the answer to #177: no dependency. You said no to a build dependency for HDF5, and I think that was right, so this writes the file directly instead. If you would rather not have the file at all, this PR is one commit and dropping it costs nothing downstream.

What this adds

raw_feature_bc_matrix.h5 and filtered_feature_bc_matrix.h5, beside the MEX directories, under --soloOutLayout CellRanger. Same counts, CellRanger's HDF5 layout: a /matrix group with the CSC triplet, the barcodes, and a /matrix/features group.

It exists because scanpy.read_10x_h5, Seurat's Read10X_h5 and CellBender's .h5 path read that single file rather than the directory.

How, given no libhdf5

src/io/hdf5.rs, 785 lines, writes the bytes. Version 0 superblock, version 1 object headers, symbol-table groups with a version 1 B-tree and a local heap, contiguous uncompressed datasets — the oldest and most widely-read variant of each structure, deliberately, since the only job here is producing files other people's readers open.

It does not chunk, compress, or write variable-length types, references, or links beyond a group's symbol table. Groups hold at most 8 entries, which is one symbol-table node; more would need node splitting and the 10x layout has at most six children in any group. Exceeding it is an error, not a silently short file.

Verified with libhdf5, not with our own reader

That distinction is the whole risk of this approach: a hand-written file can satisfy a hand-written reader and still be invalid. So the check runs HDF5's own C library. test/h5_conformance.sh on the 20 000-read fixture:

== libhdf5 opens it and walks the whole tree
   OK   h5ls -r
   OK   h5dump -H
   OK   h5stat

== libhdf5 rewrites it from scratch and finds no difference
   OK   h5repack
   OK   h5diff (no difference)

== the readers this file exists for
   scanpy    : (200, 7127) cells x genes, nnz=13959, sum=15439, first barcode AAAGAACGTCGAGCTC-1, first gene YDL246C
   cellbender: (200, 7127) cells x genes, nnz=13959, sum=15439
   OK   scanpy read_10x_h5 + CellBender loader

h5repack reads the file with libhdf5 and writes a fresh one; h5diff then finds no difference between that and ours. That is the strongest statement available short of a formal validator: the C library round-trips our bytes and agrees on every object.

The two Python readers are scanpy's _read_v3_10x_h5 and CellBender's get_matrix_from_cellranger_h5, transcribed into the script so it has no heavy imports. Both were also run against a real cellranger count 10.0.0 .h5 for comparison, and read ours the same way.

And the .h5 equals the .mtx: same 13 959 entries and 15 439 counts, compared entry by entry, for both the raw and filtered files.

What cargo test can and cannot do here

It cannot validate an HDF5 file — that needs libhdf5, which is the thing we are not depending on. So the suite locks what it can:

• the_image_is_deterministic_and_matches_the_validated_layout — two builds of the same tree are byte-identical, and the image matches a checksum taken from bytes libhdf5 accepted. If a change moves it, test/h5_conformance.sh has to be re-run before the constant is updated, because only libhdf5 can say whether the new bytes are still valid. That is written in the test.
• superblock_is_well_formed, every_allocation_is_eight_byte_aligned, symbol_table_entries_are_sorted_by_name, too_many_children_for_one_node_is_refused, an_overlong_string_is_an_error_not_a_truncation, integer_data_is_little_endian_in_element_order, strings_are_null_padded_to_the_field_width
• test_solo_out_layout_cellranger — extended: both files exist, carry the HDF5 signature, and their superblock end-of-file address equals the actual file length

This is the same shape as the differential tests: the real oracle is external and runs off CI.

Compression

Second commit. Datasets are deflated, through the libdeflater already in the tree, so still no new dependency: a version 3 chunked layout message, a version 1 filter pipeline message, and a version 1 chunk B-tree, one chunk per dataset.

	size
uncompressed (first commit)	451 KB
deflated (this PR)	88 216 bytes
CellRanger, same matrix	149 472 bytes

We end up smaller than CellRanger because it pads every string field to 256 bytes and we pad to the longest value present. Values are identical either way.

Datasets at or below 1 kB stay contiguous: a chunk B-tree node is about 2 kB allocated at full capacity, so compressing a small dataset costs more than it saves. CellRanger draws the same line, leaving _all_tag_keys uncompressed.

Two latent bugs fell out of this, and they are the argument for the h5repack check. The fill value message declared no fill value where HDF5 writes a zero-length one, and the filter pipeline carried no filter name. h5dump, h5ls, h5stat, h5py, scanpy and CellBender all accepted the result and returned correct data. h5repack rewrote it into a file h5dump then could not read. Both are fixed; the golden test now says explicitly why the conformance script has to be re-run before its constant is updated.

Where it still differs from CellRanger's file

CellRanger chunks at 65 536 elements; we use one chunk per dataset, so a reader decompresses a whole dataset to touch any of it. Every reader of these files loads them whole, and one chunk means one B-tree entry and no node splitting.

There is a warning, not a refusal, if someone pairs this layout with an explicit --soloOutRawBarcodes Whitelist: the writer holds the barcode strings in memory to size a fixed-width dataset, and a 3.7-million-column raw matrix is expensive. The layout's own default is Observed, so this needs two flags to reach.

genome carries the --genomeDir directory name, since we have no equivalent of the name given to cellranger mkref, and software_version says rustar-aligner, not a CellRanger version. Recorded in DIVERGENCE.md §3.5 with the rest.

Gate: 580 lib + 27 integration tests (plus one --ignored test that writes the golden file for the conformance script), cargo clippy --all-targets -- -D warnings, cargo fmt --check, cargo +1.89 check --all-targets, all green.

[BenjaminDEMAILLE]

Closing per the discussion in #177: an in-tree HDF5 writer adds maintenance surface to rustar for a need that is convenience rather than necessity — CellBender, scanpy and Seurat all read the MEX directory #178 produces. The offer of a standalone pure-Rust crate, hyalite-shaped and transferable to scverse, is in #177 if a real request for the format turns up. The branch keeps the work and the libhdf5 conformance script.