Fix first-run checkpoint download race and CC measurements under NumPy 2#844
Open
arvraz wants to merge 2 commits into
Open
Fix first-run checkpoint download race and CC measurements under NumPy 2#844arvraz wants to merge 2 commits into
arvraz wants to merge 2 commits into
Conversation
added 2 commits
July 14, 2026 17:10
download_checkpoint() wrote the HTTP response straight to the final path, so a
concurrent caller could observe the file through the exists() guard in
check_and_download_ckpts() while it was still being written and torch.load a
truncated checkpoint. This reliably crashes the CorpusCallosum module on a
fresh run, where the localization and segmentation models are loaded in
parallel threads that both call download_checkpoints():
RuntimeError: PytorchStreamReader failed reading file data/1: file read failed
Download to a temporary file in the same directory and os.replace() it into
place, so the checkpoint only ever becomes visible at its final path once it
is complete.
NumPy 2.0 removed the 2-element form of np.cross, which now raises
"Both input arrays must be (arrays of) 3-dimensional vectors". The corpus
callosum shape code used np.cross only to obtain the sign (z-component) of the
angle between two in-slice 2D vectors, so every slice's measurements failed:
ValueError: Both input arrays must be (arrays of) 3-dimensional vectors,
but they are 2 and 2 dimensional instead.
Compute the 2D cross-product scalar directly (a[0]*b[1] - a[1]*b[0]). This is
equivalent and works on the pinned numpy (2.4.4) as well as newer releases.
Member
|
Hi thanks for the contribution. The numpy 2.5 fix is a duplicate from here: #842 We will check the race condition. I think the same checkpoint files should not be downloaded twice, the localiser should download only its own weights and the CC segmentation its own. Alternatively, all weights should be downloaded before the parallel block starts. We should avoid duplicate parallel downloads. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two independent bugs surfaced while running the full pipeline (
run_fastsurfer.sh, seg + surface) on a fresh install. Both are fixed here; they are unrelated and can be split into separate PRs if preferred.1. Checkpoint download race →
PytorchStreamReader failed reading file data/1On a fresh run (empty
checkpoints/), the CorpusCallosum module crashes:which then cascades (
callosum.CC.orig.mgzmissing →paint_cc_into_pred.py→ noaseg.auto.mgz→ segstats fails).Cause:
download_checkpoint()inFastSurferCNN/utils/checkpoint.pywrites the response directly to the final path, whilecheck_and_download_ckpts()guards downloads withif not checkpoint_path.exists()— i.e. it checks existence, not completeness.CorpusCallosum/fastsurfer_cc.pyloads the localization and segmentation models in two parallel threads, and both calldownload_checkpoints(cc=True). One thread creates the (70 MB) file and begins writing; the other sees it "exists", skips the download, andtorch.loads the partially-written file.Fix: download to a temporary file in the same directory and
os.replace()it into place, so the checkpoint only ever appears at its final path once complete.2. Corpus callosum measurements fail under NumPy ≥ 2
Every CC slice fails with:
so all CC morphometry is silently dropped (
cc_num_failed_slices= number of slices).Cause: NumPy 2.0 removed the 2-element form of
np.cross.CorpusCallosum/shape/subsegment_contour.pyandCorpusCallosum/utils/mapping_helpers.pyuse it only to get the sign (z-component) of the angle between two in-slice 2D vectors. This affects the pinnedrequirements.txt(numpy 2.4.4) as well as newer releases.Fix: compute the 2D cross-product scalar directly,
a[0]*b[1] - a[1]*b[0](equivalent, and NumPy-version-independent).Verification
PytorchStreamReader, both checkpointstorch.loadvalid.callosum.CC.all_slices.jsonwritten withcc_num_failed_slices: 0(previously all slices failed).run_fastsurfer.sh(seg + surface) completes end-to-end with valid surfaces and thickness.ruff checkclean;test/image(387 tests) still pass.