Skip to content

A100 scoring #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aahladc merged 13 commits into from
Jul 9, 2026
Merged

A100 scoring #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
5f3bbda
Add submission with muon torch learning rate fix
aahladc Jun 25, 2026
822bbd1
Add submission with muon torch learning rate fix and forward pass wit…
aahladc Jun 25, 2026
13c31ca
Minor changes to smooth out submission scoring.
aahladc Jun 25, 2026
0dc4225
Include configs used for scoring submissions
aahladc Jun 25, 2026
1e87334
Include submission scoring logs
aahladc Jun 25, 2026
46e69f1
Add submission scoring results
aahladc Jun 25, 2026
78a586d
add plotting script and plots for the schedule free adamw training curve
wyfEmma Jun 4, 2026
c9b8879
revise the script for:
wyfEmma Jun 4, 2026
ec13723
the script now supports all algorithms and have the option to choose …
wyfEmma Jun 10, 2026
a553d78
refactor plot script to be functional so that it could be reused in c…
wyfEmma Jul 6, 2026
c5c68d5
turn the plot script into a jupiter notebook
wyfEmma Jul 9, 2026
036e67b
reorganize logs, artifacts and utils
priyakasimbeg Jul 9, 2026
0aabf28
Script that generates step size comparison for schedule free adamw ac…
wyfEmma Jun 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
The diff you're trying to view is too large. We only load the first 3000 changed files.
27 changes: 27 additions & 0 deletions artifacts/leaderboard_v1/ademamix_summary.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
,workload,trial,val target metric name,val target metric value,val target reached,best metric value on val,time to best eval on val (s),time to target on val (s),step_time (s),step_hint
7,criteo1tb_pytorch,trial_1,validation/loss,0.123735,True,0.1236847321358005,7839.193164110184,7839.193164110184,1.0010825763927422,10666
16,criteo1tb_pytorch,trial_1,validation/loss,0.123735,False,0.1242111736866617,13214.635874271393,inf,1.0010825763927422,10666
24,criteo1tb_pytorch,trial_1,validation/loss,0.123735,True,0.1236501955610997,7835.540827274322,7479.550709962845,1.0010825763927422,10666
1,fastmri_pytorch,trial_1,validation/ssim,0.723653,True,0.7248104441078714,1311.1073644161224,1311.1073644161224,0.9420394021221715,18094
10,fastmri_pytorch,trial_1,validation/ssim,0.723653,True,0.7249505124067952,1517.7321906089785,1517.7321906089785,0.9420394021221715,18094
19,fastmri_pytorch,trial_1,validation/ssim,0.723653,True,0.724893770663337,1555.3687834739685,1555.3687834739685,0.9420394021221715,18094
8,finewebedu_lm_pytorch,trial_1,validation/ppl,22.432,True,22.14082852319187,18025.636202573776,18025.636202573776,0.40403201664468946,72000
17,finewebedu_lm_pytorch,trial_1,validation/ppl,22.432,True,22.23496029572029,18027.61917901039,18027.61917901039,0.40403201664468946,72000
25,finewebedu_lm_pytorch,trial_1,validation/ppl,22.432,True,22.247438140097906,18025.89689064026,18025.89689064026,0.40403201664468946,72000
0,imagenet_resnet_pytorch,trial_1,validation/accuracy,0.77431,False,0.60218,59997.68517279625,inf,0.6740027356507963,195999
9,imagenet_resnet_pytorch,trial_1,validation/accuracy,0.77431,False,0.61658,73926.78414797783,inf,0.6740027356507963,195999
18,imagenet_resnet_pytorch,trial_1,validation/accuracy,0.77431,False,0.61518,73948.011013031,inf,0.6740027356507963,195999
2,imagenet_vit_pytorch,trial_1,validation/accuracy,0.77309,False,0.61544,95268.04540586472,inf,0.802424341129202,167999
11,imagenet_vit_pytorch,trial_1,validation/accuracy,0.77309,False,0.61512,92620.5374662876,inf,0.802424341129202,167999
3,librispeech_conformer_pytorch,trial_1,validation/wer,0.085884,True,0.0848356104861681,54136.87689781189,54136.87689781189,0.7305170534584913,76000
12,librispeech_conformer_pytorch,trial_1,validation/wer,0.085884,True,0.0838990006276251,48894.85426735878,48894.85426735878,0.7305170534584913,76000
20,librispeech_conformer_pytorch,trial_1,validation/wer,0.085884,True,0.0842659199536522,48888.971920251846,48888.971920251846,0.7305170534584913,76000
5,librispeech_deepspeech_pytorch,trial_1,validation/wer,0.119936,False,0.1200695215565104,34737.411699295044,inf,0.7884586655586601,38400
14,librispeech_deepspeech_pytorch,trial_1,validation/wer,0.119936,True,0.1185245980785014,28953.85911822319,28953.85911822319,0.7884586655586601,38400
22,librispeech_deepspeech_pytorch,trial_1,validation/wer,0.119936,False,0.1215758219475691,33312.74022984505,inf,0.7884586655586601,38400
4,ogbg_pytorch,trial_1,validation/mean_average_precision,0.28098,True,0.2872654622003903,7232.420216321945,6329.966245889664,0.2304398308491926,52000
13,ogbg_pytorch,trial_1,validation/mean_average_precision,0.28098,True,0.2920392515125629,8137.10068655014,6332.58651471138,0.2304398308491926,52000
21,ogbg_pytorch,trial_1,validation/mean_average_precision,0.28098,True,0.283304967314845,5891.7984273433685,5891.7984273433685,0.2304398308491926,52000
6,wmt_pytorch,trial_1,validation/bleu,30.8491,True,30.87605698677609,17399.639942407608,17399.639942407608,0.1538000967767499,120000
15,wmt_pytorch,trial_1,validation/bleu,30.8491,False,30.80807424915632,19338.09857201576,inf,0.1538000967767499,120000
23,wmt_pytorch,trial_1,validation/bleu,30.8491,True,30.867059390411267,16752.356283187866,16752.356283187866,0.1538000967767499,120000
26 changes: 26 additions & 0 deletions artifacts/leaderboard_v1/cautious_nadamw_summary.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
,workload,trial,val target metric name,val target metric value,val target reached,best metric value on val,time to best eval on val (s),time to target on val (s),step_time (s),step_hint
7,criteo1tb_jax,trial_1,validation/loss,0.123735,False,0.1256997472383533,2863.6579899787903,inf,0.9266347767470695,10666
15,criteo1tb_jax,trial_1,validation/loss,0.123735,False,0.1258371449566668,1792.0047852993011,inf,0.9266347767470695,10666
24,criteo1tb_jax,trial_1,validation/loss,0.123735,False,0.1246944847459977,9989.461620092392,inf,0.9266347767470695,10666
1,fastmri_jax,trial_1,validation/ssim,0.723653,True,0.7244385315181837,1608.188658952713,1608.188658952713,0.6559752236253,18094
9,fastmri_jax,trial_1,validation/ssim,0.723653,True,0.7238933024409117,1383.1596400737762,1383.1596400737762,0.6559752236253,18094
17,fastmri_jax,trial_1,validation/ssim,0.723653,True,0.7236819978369443,2571.815328359604,2571.815328359604,0.6559752236253,18094
23,finewebedu_lm_jax,trial_1,validation/ppl,22.432,True,21.63268571712524,25738.981688261032,25738.981688261032,0.44144648333990966,72000
0,imagenet_resnet_jax,trial_1,validation/accuracy,0.77431,False,0.7466,69885.5102982521,inf,0.25149491638964216,195999
8,imagenet_resnet_jax,trial_1,validation/accuracy,0.77431,False,0.74748,67896.18414497375,inf,0.25149491638964216,195999
16,imagenet_resnet_jax,trial_1,validation/accuracy,0.77431,False,0.74964,61917.01934456825,inf,0.25149491638964216,195999
6,imagenet_vit_jax,trial_1,validation/accuracy,0.77309,False,0.7506,87452.66640520096,inf,0.49596923795226744,167999
14,imagenet_vit_jax,trial_1,validation/accuracy,0.77309,False,0.75158,82308.66560220718,inf,0.49596923795226744,167999
22,imagenet_vit_jax,trial_1,validation/accuracy,0.77309,False,0.75034,87448.36735343933,inf,0.49596923795226744,167999
4,librispeech_conformer_jax,trial_1,validation/wer,0.085884,False,0.1421619272920388,64988.59479641914,inf,2.589838852085477,76000
12,librispeech_conformer_jax,trial_1,validation/wer,0.085884,False,0.1426254043354415,65012.818663835526,inf,2.589838852085477,76000
20,librispeech_conformer_jax,trial_1,validation/wer,0.085884,False,0.1444986240525274,64979.35094237328,inf,2.589838852085477,76000
5,librispeech_deepspeech_jax,trial_1,validation/wer,0.119936,False,0.2943175783324482,10338.725378990172,inf,2.7988120351912658,38400
13,librispeech_deepspeech_jax,trial_1,validation/wer,0.119936,False,0.2872881765075074,10334.769649744034,inf,2.7988120351912658,38400
21,librispeech_deepspeech_jax,trial_1,validation/wer,0.119936,False,0.2853473663882586,14680.395282268524,inf,2.7988120351912658,38400
3,ogbg_jax,trial_1,validation/mean_average_precision,0.28098,True,0.2845644139519467,5444.764040708542,4992.686820983887,0.1979359465075458,52000
11,ogbg_jax,trial_1,validation/mean_average_precision,0.28098,True,0.2874222775151546,5446.810508966446,4542.640103340149,0.1979359465075458,52000
19,ogbg_jax,trial_1,validation/mean_average_precision,0.28098,True,0.2809900831846614,4995.647005558014,4995.647005558014,0.1979359465075458,52000
2,wmt_jax,trial_1,validation/bleu,30.8491,True,30.876108999624343,12943.72738480568,12943.72738480568,0.1255462995675322,120000
10,wmt_jax,trial_1,validation/bleu,30.8491,True,31.02074381397432,14876.415894269943,14232.458706855774,0.1255462995675322,120000
18,wmt_jax,trial_1,validation/bleu,30.8491,True,30.91251520933702,15517.656443834305,15517.656443834305,0.1255462995675322,120000
28 changes: 28 additions & 0 deletions artifacts/leaderboard_v1/lion_summary.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
,workload,trial,val target metric name,val target metric value,val target reached,best metric value on val,time to best eval on val (s),time to target on val (s),step_time (s),step_hint
7,criteo1tb_pytorch,trial_1,validation/loss,0.123735,False,0.1241905161220999,9978.416061878204,inf,0.91592558011244,10666
16,criteo1tb_pytorch,trial_1,validation/loss,0.123735,False,0.1242616643239586,12466.455169200895,inf,0.91592558011244,10666
25,criteo1tb_pytorch,trial_1,validation/loss,0.123735,False,0.1242412303757036,12106.893656492231,inf,0.91592558011244,10666
1,fastmri_pytorch,trial_1,validation/ssim,0.723653,True,0.7238031751195836,619.9493722915649,619.9493722915649,0.27376442936493867,18094
10,fastmri_pytorch,trial_1,validation/ssim,0.723653,True,0.7243804158826322,1421.2225756645205,1421.2225756645205,0.27376442936493867,18094
19,fastmri_pytorch,trial_1,validation/ssim,0.723653,True,0.7239882383845667,1394.1750195026398,1394.1750195026398,0.27376442936493867,18094
8,finewebedu_lm_pytorch,trial_1,validation/ppl,22.432,True,21.261897266260053,23164.569310188293,23164.569310188293,0.38839825067830164,72000
17,finewebedu_lm_pytorch,trial_1,validation/ppl,22.432,True,21.110008038411504,23177.09142231941,23177.09142231941,0.38839825067830164,72000
26,finewebedu_lm_pytorch,trial_1,validation/ppl,22.432,True,21.130699319951358,23172.039316177368,23172.039316177368,0.38839825067830164,72000
0,imagenet_resnet_pytorch,trial_1,validation/accuracy,0.77431,False,0.61338,73980.61954474449,inf,0.6743097256486612,195999
9,imagenet_resnet_pytorch,trial_1,validation/accuracy,0.77431,False,0.61892,67952.96683573723,inf,0.6743097256486612,195999
18,imagenet_resnet_pytorch,trial_1,validation/accuracy,0.77431,False,0.62902,67951.73512744904,inf,0.6743097256486612,195999
2,imagenet_vit_pytorch,trial_1,validation/accuracy,0.77309,False,0.6242,87448.87829089165,inf,0.8120775926593171,167999
11,imagenet_vit_pytorch,trial_1,validation/accuracy,0.77309,False,0.63292,95191.34969496728,inf,0.8120775926593171,167999
20,imagenet_vit_pytorch,trial_1,validation/accuracy,0.77309,False,0.6418,95165.0220131874,inf,0.8120775926593171,167999
3,librispeech_conformer_pytorch,trial_1,validation/wer,0.085884,False,0.1153188818616327,57621.20974063873,inf,0.7231872968606753,76000
12,librispeech_conformer_pytorch,trial_1,validation/wer,0.085884,False,0.1507845314536764,54127.30689024925,inf,0.7231872968606753,76000
21,librispeech_conformer_pytorch,trial_1,validation/wer,0.085884,False,0.1264423309032974,57621.14528346062,inf,0.7231872968606753,76000
5,librispeech_deepspeech_pytorch,trial_1,validation/wer,0.119936,False,0.1247042919905373,31862.61851024628,inf,0.7694697594617248,38400
14,librispeech_deepspeech_pytorch,trial_1,validation/wer,0.119936,False,0.126799594457587,31838.147053956985,inf,0.7694697594617248,38400
23,librispeech_deepspeech_pytorch,trial_1,validation/wer,0.119936,False,0.1238063052189446,31852.81674218177,inf,0.7694697594617248,38400
4,ogbg_pytorch,trial_1,validation/mean_average_precision,0.28098,True,0.2857323411413317,7223.987207174301,7223.987207174301,0.21286914966307793,52000
13,ogbg_pytorch,trial_1,validation/mean_average_precision,0.28098,True,0.2812007204404579,6782.790376663208,6782.790376663208,0.21286914966307793,52000
22,ogbg_pytorch,trial_1,validation/mean_average_precision,0.28098,True,0.283014990979593,8582.123725652695,8582.123725652695,0.21286914966307793,52000
6,wmt_pytorch,trial_1,validation/bleu,30.8491,True,30.948023339872734,15475.552730083466,15475.552730083466,0.1413890021270408,120000
15,wmt_pytorch,trial_1,validation/bleu,30.8491,True,30.87531810470684,17401.493114233017,15474.955520391464,0.1413890021270408,120000
24,wmt_pytorch,trial_1,validation/bleu,30.8491,False,30.76188896433939,17388.50763297081,inf,0.1413890021270408,120000
28 changes: 28 additions & 0 deletions artifacts/leaderboard_v1/muon_summary.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
,workload,trial,val target metric name,val target metric value,val target reached,best metric value on val,time to best eval on val (s),time to target on val (s),step_time (s),step_hint
8,criteo1tb_jax,trial_1,validation/loss,0.123735,False,0.1245190224490917,13217.0788064003,inf,1.80744596371582,10666
17,criteo1tb_jax,trial_1,validation/loss,0.123735,False,0.1246482386019362,12154.283058166504,inf,1.80744596371582,10666
26,criteo1tb_jax,trial_1,validation/loss,0.123735,False,0.1245746128447548,13221.646473169329,inf,1.80744596371582,10666
1,fastmri_jax,trial_1,validation/ssim,0.723653,True,0.7239323209763646,1145.923245429993,1145.923245429993,0.15300968625117742,18094
10,fastmri_jax,trial_1,validation/ssim,0.723653,True,0.7238764035681626,2594.490219116211,2594.490219116211,0.15300968625117742,18094
19,fastmri_jax,trial_1,validation/ssim,0.723653,True,0.7247154394696117,1869.765276670456,1869.765276670456,0.15300968625117742,18094
7,finewebedu_lm_jax,trial_1,validation/ppl,22.432,True,21.808373188667463,20594.037749767303,20594.037749767303,0.504613462023225,72000
16,finewebedu_lm_jax,trial_1,validation/ppl,22.432,True,21.85239511139204,20594.19531917572,20594.19531917572,0.504613462023225,72000
25,finewebedu_lm_jax,trial_1,validation/ppl,22.432,True,21.82093364841247,20593.710190296173,20593.710190296173,0.504613462023225,72000
0,imagenet_resnet_jax,trial_1,validation/accuracy,0.77431,False,0.73021996,73908.46513557434,inf,0.34063982264727855,195999
9,imagenet_resnet_jax,trial_1,validation/accuracy,0.77431,False,0.73056,53931.644364357,inf,0.34063982264727855,195999
18,imagenet_resnet_jax,trial_1,validation/accuracy,0.77431,False,0.71808,65941.6421945095,inf,0.34063982264727855,195999
6,imagenet_vit_jax,trial_1,validation/accuracy,0.77309,True,0.77518,77162.91657114029,77162.91657114029,0.5145936103030498,167999
15,imagenet_vit_jax,trial_1,validation/accuracy,0.77309,False,0.75193995,95183.25943279266,inf,0.5145936103030498,167999
24,imagenet_vit_jax,trial_1,validation/accuracy,0.77309,True,0.77476,74592.82279109955,74592.82279109955,0.5145936103030498,167999
4,librispeech_conformer_jax,trial_1,validation/wer,0.085884,False,0.1673441799835852,64973.90770673752,inf,2.486110555855915,76000
13,librispeech_conformer_jax,trial_1,validation/wer,0.085884,False,0.1665524067011055,64973.39529895783,inf,2.486110555855915,76000
22,librispeech_conformer_jax,trial_1,validation/wer,0.085884,False,0.1644281369188432,64948.560970783234,inf,2.486110555855915,76000
5,librispeech_deepspeech_jax,trial_1,validation/wer,0.119936,False,0.174721189591078,55211.92119574547,inf,2.709749883580429,38400
14,librispeech_deepspeech_jax,trial_1,validation/wer,0.119936,False,0.1775792980253946,55245.24317121506,inf,2.709749883580429,38400
23,librispeech_deepspeech_jax,trial_1,validation/wer,0.119936,False,0.1763819823299377,53775.95950937271,inf,2.709749883580429,38400
3,ogbg_jax,trial_1,validation/mean_average_precision,0.28098,False,0.279643299276242,1828.2280414104464,inf,0.19192375124386668,52000
12,ogbg_jax,trial_1,validation/mean_average_precision,0.28098,False,0.2754040150725418,1378.5888230800629,inf,0.19192375124386668,52000
21,ogbg_jax,trial_1,validation/mean_average_precision,0.28098,False,0.2761794983317486,1827.266467809677,inf,0.19192375124386668,52000
2,wmt_jax,trial_1,validation/bleu,30.8491,False,28.997438769958425,12296.452765226364,inf,0.173934285073483,120000
11,wmt_jax,trial_1,validation/bleu,30.8491,False,28.65403612242089,11011.619378566742,inf,0.173934285073483,120000
20,wmt_jax,trial_1,validation/bleu,30.8491,False,28.777579127157672,12941.239448785782,inf,0.173934285073483,120000
25 changes: 25 additions & 0 deletions artifacts/leaderboard_v1/muon_torch_jax_hps_achandr_summary.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
,workload,trial,val target metric name,val target metric value,val target reached,best metric value on val,time to best eval on val (s),time to target on val (s),step_time (s),step_hint
6,criteo1tb_pytorch,trial_1,validation/loss,0.123735,False,0.1244582424910858,9634.312291383743,inf,0.9711783830686549,10666
14,criteo1tb_pytorch,trial_1,validation/loss,0.123735,False,0.1244297249629434,10347.2724711895,inf,0.9711783830686549,10666
22,criteo1tb_pytorch,trial_1,validation/loss,0.123735,False,0.1244362416603791,8941.123232603073,inf,0.9711783830686549,10666
1,fastmri_pytorch,trial_1,validation/ssim,0.723653,True,0.7239354122335748,1263.3804235458374,1263.3804235458374,1.1274927614012604,18094
9,fastmri_pytorch,trial_1,validation/ssim,0.723653,True,0.7244227317591094,1397.0868299007416,1397.0868299007416,1.1274927614012604,18094
17,fastmri_pytorch,trial_1,validation/ssim,0.723653,True,0.7251363313124296,1357.4687869548798,1357.4687869548798,1.1274927614012604,18094
7,finewebedu_lm_pytorch,trial_1,validation/ppl,22.432,True,20.17356939052113,10351.42058467865,10351.42058467865,0.2033421407507006,72000
15,finewebedu_lm_pytorch,trial_1,validation/ppl,22.432,True,20.13693460840124,10338.734079360962,10338.734079360962,0.2033421407507006,72000
23,finewebedu_lm_pytorch,trial_1,validation/ppl,22.432,True,20.28190476358841,10335.659363031387,10335.659363031387,0.2033421407507006,72000
0,imagenet_resnet_pytorch,trial_1,validation/accuracy,0.77431,False,0.636,67946.58710741997,inf,0.733399593865325,195999
8,imagenet_resnet_pytorch,trial_1,validation/accuracy,0.77431,False,0.63268,69927.26937556267,inf,0.733399593865325,195999
16,imagenet_resnet_pytorch,trial_1,validation/accuracy,0.77431,False,0.62592,73906.59245371819,inf,0.733399593865325,195999
2,imagenet_vit_pytorch,trial_1,validation/accuracy,0.77309,False,0.64002,95158.52423453332,inf,0.8253189693677481,167999
10,imagenet_vit_pytorch,trial_1,validation/accuracy,0.77309,False,0.6509,95160.28095626833,inf,0.8253189693677481,167999
18,imagenet_vit_pytorch,trial_1,validation/accuracy,0.77309,False,0.64674,95137.99645900726,inf,0.8253189693677481,167999
4,librispeech_deepspeech_pytorch,trial_1,validation/wer,0.119936,False,0.1233911070342297,28965.4743244648,inf,0.7700204282210652,38400
12,librispeech_deepspeech_pytorch,trial_1,validation/wer,0.119936,False,0.1235262878385555,30396.61142873764,inf,0.7700204282210652,38400
20,librispeech_deepspeech_pytorch,trial_1,validation/wer,0.119936,False,0.1276493023704919,27513.07432436943,inf,0.7700204282210652,38400
3,ogbg_pytorch,trial_1,validation/mean_average_precision,0.28098,False,0.2792997481830299,1838.9561376571653,inf,0.22362731819104265,52000
11,ogbg_pytorch,trial_1,validation/mean_average_precision,0.28098,False,0.2783637947593863,2305.59510588646,inf,0.22362731819104265,52000
19,ogbg_pytorch,trial_1,validation/mean_average_precision,0.28098,False,0.276606606902414,1834.394849538803,inf,0.22362731819104265,52000
5,wmt_pytorch,trial_1,validation/bleu,30.8491,False,28.48839040257156,6481.575395107269,inf,0.0893746101765593,120000
13,wmt_pytorch,trial_1,validation/bleu,30.8491,False,28.84975754265453,5198.2468848228455,inf,0.0893746101765593,120000
21,wmt_pytorch,trial_1,validation/bleu,30.8491,False,28.563567621183555,5214.693088293076,inf,0.0893746101765593,120000
Loading
Loading