[algoperf] Allow scoring with target relaxations - #48
Open
andyliu14 wants to merge 6 commits into
Open
Conversation
andyliu14
commented
Jul 17, 2026
andyliu14
commented
Contributor
- Enables seeing how "close to convergence" changes scoring
- Makes min/max obj declaration at config-time rather than naming-based
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
1. Enables seeing how "close to convergence" changes scoring 2. Makes min/max obj declaration at config-time rather than naming-based
andyliu14
force-pushed
the
aliu_relaxation_scoring
branch
from
July 17, 2026 04:57
015330a to
d49be56
Compare
`uv sync --extra dev` now reproduces the exact scoring environment from the committed lockfile. The dev extra stays in optional-dependencies so CI's `pip install -e .[dev]` keeps working. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Forks the v1 colab notebook into a v2 that runs locally against the repo's self-contained scoring package: repo-root bootstrap, repo-relative paths, helpers imported from scoring.score_submissions instead of mirrored, and a WorkloadConfig passed through, so scores match python -m scoring.score_submissions exactly. leaderboard_v1 was generated against the pre-vendoring algorithmic-efficiency scoring code, whose hardcoded NUM_BASE_WORKLOADS = 8 predates the fineweb-edu workload; with 9 workloads scored and imagenet_resnet unsolved by all submissions, every score was inflated by exactly 9/8. leaderboard_v2 regenerates all artifacts with the config-driven 9-workload denominator: rankings are unchanged, scores scale by 8/9, and time_to_target_table.tex is byte-identical to v1 (time-to-target values are denominator-independent). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scores all submissions with the identical performance-profile machinery using global_step instead of wall-clock seconds (artifact suffix _steps), and emits a steps-to-target table, a wall-clock vs step-based leaderboard comparison table, and a scatter of the two scores per submission. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One marker per algorithm family with colorblind-safe colors cycling within each family, family-grouped legend beside the plot, and print-true sizing (built at the 6.5in textwidth it is included at, with paper point sizes). Artifacts regenerated by re-executing the notebook. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Compares the three paired algorithms (Schedule-Free AdamW v1/v2, Muon) on per-example step time and time to target; emits the framework_comparison heatmap artifacts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.