Weight snapshot download progress by file size, not file count - #369
Open
atlascodesai wants to merge 1 commit into
Open
Weight snapshot download progress by file size, not file count#369atlascodesai wants to merge 1 commit into
atlascodesai wants to merge 1 commit into
Conversation
HubApi.snapshot aggregates per-file progress into a parent Progress whose totalUnitCount is the number of files, giving every file equal weight. For a typical model repo (one multi-GB weight shard plus several tiny JSON/config files) the parent fractionCompleted is dominated by file count, so the bar barely moves while the large shard downloads and then jumps near the end — even though the per-file downloader already tracks real bytes. Weight each file by its remote byte size instead: - Pre-fetch each file's FileMetadata once (reusing the downloader's own source URL, so repo-type handling stays correct) and size its child Progress by bytes; the parent totalUnitCount is the sum of byte sizes. - Thread that metadata into HubFileDownloader.remoteMetadata so download() reuses it instead of issuing a second getFileMetadata request per file (no extra network round-trips vs. before). - Files with unknown size fall back to a minimal weight so they still advance progress without dominating it. Offline mode is unaffected (it returns before this path). Snapshot tests now assert completedUnitCount reaches totalUnitCount rather than a hard-coded file count, which is correct regardless of the weighting scheme.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
HubApi.snapshotaggregates per-file progress into a parentProgresswhosetotalUnitCountis the number of files, so every file weighs equally. For a typical model repo (one multi-GB weight shard + several tiny JSON/config files), the aggregatefractionCompletedbarely moves while the large shard downloads and then jumps near the end — even though the per-file downloader already tracks real bytes.Change
Weight each file by its remote byte size instead:
FileMetadataonce (reusing the downloader's ownsourceURL, so non-model repo-type prefixes stay correct) and size its childProgressby bytes; the parenttotalUnitCountis the sum of byte sizes.HubFileDownloader.remoteMetadatasodownload()reuses it instead of issuing a secondgetFileMetadatarequest per file — no extra network round-trips vs. before.Tests
completedUnitCountreachestotalUnitCount(correct regardless of weighting scheme).testSnapshotProgressIsByteWeightedasserts the aggregatetotalUnitCountequals the summed remote byte sizes.SnapshotDownloadTests— 12 tests — pass against the live Hugging Face Hub (real downloads from public repos), including large-file, background, revision/PR-revision, and metadata cases.AI disclosure: this change was written by Anthropic's Claude (Opus 4.8) and independently reviewed by OpenAI's Codex (GPT‑5.5). It was verified by running the full
SnapshotDownloadTestssuite (12 tests, incl. the new byte-weighting test) against the live Hugging Face Hub, and by building theHuband test targets againstmain.