Skip to content

Fix first-run checkpoint download race and CC measurements under NumPy 2#844

Open
arvraz wants to merge 2 commits into
Deep-MI:devfrom
arvraz:fix/checkpoint-race-and-numpy2-cross
Open

Fix first-run checkpoint download race and CC measurements under NumPy 2#844
arvraz wants to merge 2 commits into
Deep-MI:devfrom
arvraz:fix/checkpoint-race-and-numpy2-cross

Conversation

@arvraz

@arvraz arvraz commented Jul 14, 2026

Copy link
Copy Markdown

Summary

Two independent bugs surfaced while running the full pipeline (run_fastsurfer.sh, seg + surface) on a fresh install. Both are fixed here; they are unrelated and can be split into separate PRs if preferred.

1. Checkpoint download race → PytorchStreamReader failed reading file data/1

On a fresh run (empty checkpoints/), the CorpusCallosum module crashes:

RuntimeError: PytorchStreamReader failed reading file data/1: file read failed

which then cascades (callosum.CC.orig.mgz missing → paint_cc_into_pred.py → no aseg.auto.mgz → segstats fails).

Cause: download_checkpoint() in FastSurferCNN/utils/checkpoint.py writes the response directly to the final path, while check_and_download_ckpts() guards downloads with if not checkpoint_path.exists() — i.e. it checks existence, not completeness. CorpusCallosum/fastsurfer_cc.py loads the localization and segmentation models in two parallel threads, and both call download_checkpoints(cc=True). One thread creates the (70 MB) file and begins writing; the other sees it "exists", skips the download, and torch.loads the partially-written file.

Fix: download to a temporary file in the same directory and os.replace() it into place, so the checkpoint only ever appears at its final path once complete.

2. Corpus callosum measurements fail under NumPy ≥ 2

Every CC slice fails with:

ValueError: Both input arrays must be (arrays of) 3-dimensional vectors,
but they are 2 and 2 dimensional instead.

so all CC morphometry is silently dropped (cc_num_failed_slices = number of slices).

Cause: NumPy 2.0 removed the 2-element form of np.cross. CorpusCallosum/shape/subsegment_contour.py and CorpusCallosum/utils/mapping_helpers.py use it only to get the sign (z-component) of the angle between two in-slice 2D vectors. This affects the pinned requirements.txt (numpy 2.4.4) as well as newer releases.

Fix: compute the 2D cross-product scalar directly, a[0]*b[1] - a[1]*b[0] (equivalent, and NumPy-version-independent).

Verification

  • Deleted the CC checkpoints and re-ran the CC module: fresh concurrent download succeeds, no PytorchStreamReader, both checkpoints torch.load valid.
  • CC measurements now compute: callosum.CC.all_slices.json written with cc_num_failed_slices: 0 (previously all slices failed).
  • Full run_fastsurfer.sh (seg + surface) completes end-to-end with valid surfaces and thickness.
  • ruff check clean; test/image (387 tests) still pass.

Arvin Razavi added 2 commits July 14, 2026 17:10
download_checkpoint() wrote the HTTP response straight to the final path, so a
concurrent caller could observe the file through the exists() guard in
check_and_download_ckpts() while it was still being written and torch.load a
truncated checkpoint. This reliably crashes the CorpusCallosum module on a
fresh run, where the localization and segmentation models are loaded in
parallel threads that both call download_checkpoints():

    RuntimeError: PytorchStreamReader failed reading file data/1: file read failed

Download to a temporary file in the same directory and os.replace() it into
place, so the checkpoint only ever becomes visible at its final path once it
is complete.
NumPy 2.0 removed the 2-element form of np.cross, which now raises
"Both input arrays must be (arrays of) 3-dimensional vectors". The corpus
callosum shape code used np.cross only to obtain the sign (z-component) of the
angle between two in-slice 2D vectors, so every slice's measurements failed:

    ValueError: Both input arrays must be (arrays of) 3-dimensional vectors,
    but they are 2 and 2 dimensional instead.

Compute the 2D cross-product scalar directly (a[0]*b[1] - a[1]*b[0]). This is
equivalent and works on the pinned numpy (2.4.4) as well as newer releases.
@m-reuter

Copy link
Copy Markdown
Member

Hi thanks for the contribution.

The numpy 2.5 fix is a duplicate from here: #842

We will check the race condition. I think the same checkpoint files should not be downloaded twice, the localiser should download only its own weights and the CC segmentation its own. Alternatively, all weights should be downloaded before the parallel block starts. We should avoid duplicate parallel downloads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants