From 3816a59a03d6fdc312e675b3efc345d7b5d572fc Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Wed, 29 Jul 2026 15:26:43 -0700 Subject: [PATCH 1/8] Refresh the YOLO baseline record: durable weights, ckpt ceiling, fork (#51) The committed record had drifted from the runs in three ways, one of them an actively false statement. "Where the weights live" said `_TODO -- stage to Hugging Face or lab storage_`. The weights are now staged: /gscratch/makelab/jonf/rampnet_yolo_baseline_51/, five arms, best.pt + last.pt + curves + a MANIFEST.md, all ten weight files sha256-verified against source. They had existed only on /gscratch/scrubbed, which auto-purges after ~21 idle days, and the run dirs date from 07-26. A reader of the old text would reasonably conclude the weights were still at risk. The status table was the 07-28 snapshot (epochs 14/10/3/3/7) and is now live (18/13/9/3/3 plus the fork). More consequentially, "every benchmark number from these checkpoints is a lower bound" is no longer uniform: y26_pano's best.pt is epoch 18 -- proven byte-identical to last.pt, sha256 aac93e74..50bf -- while the other four are frozen at epoch 1. Reporting all five the same way would both understate y26_pano and overstate the rest, so the caveat is now per-arm and says "one-epoch model" in those words where that is what it is. Two findings were missing entirely. The ckpt slice ceiling is a limitation distinct from the LR instability already documented: across 158 scheduling segments the longest ever granted was 8.24 h and none reached 9.5 h, while tiles epoch time rose from ~5.2 h to ~9.5 h under filesystem contention (the logs warn `read: 2.1+-1.3 MB/s` over 557,413 tiles). Since Ultralytics checkpoints only at epoch boundaries, an epoch longer than the slice can never complete -- so both tiles arms burned ~19 GPU-hours each and logged zero epochs. That is a structural gap, not a slow one, and it is the same mechanism that dropped y11x_tiles. The response was additive: y26_tiles forked onto one of makelab's two gpu-l40s GPUs (job 37889646, 72 h cap), original left running on ckpt. The fork's provenance is recorded in full because it is easy to get silently wrong -- resume=True restores every arg from the checkpoint including save_dir, so the copied last.pt needs project/name/save_dir/model/resume rewritten or the fork resumes into the live original directory and corrupts a training arm. Everything governing the schedule was asserted unchanged on read-back. Deferred deliberately: re-pulling runs/*/results.csv, regenerating figures/, and updating the docs/model_comparison.md paragraph -- all worth doing once against final curves rather than a moving target. The file now says so explicitly so the staleness reads as a decision rather than an oversight. Co-Authored-By: Claude Opus 5 (1M context) --- .../model_comparison/yolo_baseline/README.md | 170 ++++++++++++++++-- 1 file changed, 156 insertions(+), 14 deletions(-) diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md index 578cc76..b06e00d 100644 --- a/scripts/model_comparison/yolo_baseline/README.md +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -5,7 +5,11 @@ trained on the RampNet dataset on Hyak (klone). It exists so the experiment surv `/gscratch/scrubbed`'s ~21-day auto-purge and is reproducible/defensible for the paper. > **Status:** the runs are in-flight (issue #51); live status is tracked there and the -> finding is summarized in `docs/model_comparison.md`. The training code lives at its +> finding is summarized in `docs/model_comparison.md` — though that summary predates the +> 2026-07-29 findings below (the ckpt slice ceiling, the per-arm `best.pt` split, the fork) +> and is deliberately left to be updated together with the final curves. **This file is the +> current record; `docs/model_comparison.md` is behind it by design, not by oversight.** The +> training code lives at its > canonical paths — `scripts/model_comparison/run_yolo_train.slurm` and the repo-root > `hyak_yolo_runbook.sh` — merged in PR #76. (This record briefly carried as-run > snapshots of both; the only drift from the canonical copies was a display-only @@ -50,20 +54,40 @@ Batches are the as-run values from each run's committed `args.yaml`. `y11x_tiles submitted at batch 3, then resubmitted at batch 12 trying to finish an epoch inside the ckpt scheduling slice (its `args.yaml` is the batch-12 attempt); neither completed one. -## Status & findings (snapshot: 2026-07-28, training in progress) +There is a seventh **run directory** on scratch, `y26_tiles_l40s`, but it is **not a seventh +config**: it is `y26_tiles` itself, resumed from its own epoch-3 checkpoint on a different +partition with every hyperparameter unchanged. See "The `y26_tiles_l40s` fork" below. + +## Status & findings (live check: 2026-07-29 ~14:45 PT, training in progress) Val-split proxy mAP50 from `results.csv`. **Every config peaks at epoch 1, collapses at epoch 3, and recovers as the learning rate decays.** The instability is universal, not per-config — see the figures. -| Config | Epochs | best (ep) | Current mAP50 | Assessment | -|--------------|--------|-----------|---------------|------------| -| `y26_pano` | 14 | 0.738 (1) | **0.659 ↑** | ✅ fully round-tripped: 0.738 → 0.125 @ep6 → 0.659 @ep14, climbing | -| `y11l_pano` | 10 | 0.779 (1) | **0.183 ↑** | 🟡 recovering — 7 epochs near 0, then 0.005 → 0.183 @ep10 | -| `y26_tiles` | 3 | 0.647 (1) | 0.280 ↓ | 🟡 in the dip, too early to call | -| `y11l_tiles` | 3 | 0.655 (1) | 0.042 ↓ | 🟡 in the dip, too early to call | -| `y11x_pano` | 7 | 0.777 (1) | **0.000** | ❌ five straight epochs at literal 0; no recovery yet | -| `y11x_tiles` | 0 | — | — | ⏹ dropped 2026-07-27 (GPU-saturated: epoch ~10 h > ckpt slice) | +> **The numbers in this table are ahead of the committed `runs/*/results.csv` and +> `figures/*.png`**, which are still the 2026-07-28 snapshot (e.g. `y26_pano` has 14 rows +> there vs 18 epochs below). The CSVs and figures will be re-pulled and regenerated once +> the arms stop, so the record ends up internally consistent against *final* curves rather +> than a moving target. Until then, prefer this table for status and the CSVs for the +> per-epoch shape of the collapse. + +| Config | Epochs | best-val (ep) | Latest mAP50 | `best.pt` holds | Assessment | +|------------------|--------|---------------|--------------|-----------------|------------| +| `y26_pano` | 18 | 0.738 (1) | **0.717 ↑** | **ep18 — current** | ✅ round-tripped 0.738 → 0.125 @ep6 → 0.717 @ep18; the only arm whose `best.pt` has cleared ep1 | +| `y11l_pano` | 13 | 0.779 (1) | **0.727 ↑** | ep1 | 🟡 recovering hard — 0.000 @ep4–5 → 0.183 @ep10 → 0.727 @ep13; ~1–2 epochs from unfreezing `best.pt` | +| `y11x_pano` | 9 | 0.777 (1) | 0.061 ↑ | ep1 | ❌ five epochs at literal 0; `patience=20` retires it ~ep22, almost certainly still holding ep1 | +| `y26_tiles` | 3 | 0.647 (1) | 0.280 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28; forked to `gpu-l40s` (below) | +| `y11l_tiles` | 3 | 0.655 (1) | 0.042 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28 | +| `y26_tiles_l40s` | 3 (+) | inherited | — | inherited ep1 | 🆕 fork of `y26_tiles` resumed at ep4 on `gpu-l40s`, ~4.1 h/epoch (see "The ckpt slice ceiling") | +| `y11x_tiles` | 0 | — | — | — | ⏹ dropped 2026-07-27 (GPU-saturated: epoch ~10 h > ckpt slice) | + +**`y26_pano`'s `best.pt` is epoch 18, not epoch 1.** Its `best.pt` and `last.pt` are +byte-identical (both sha256 `aac93e74…50bf`), which is direct evidence rather than an +inference from mtimes: mAP50-95 0.460 @ep18 vs 0.413 @ep1. The other four arms' `best.pt` +files are frozen at **epoch 1** — mtimes 2026-07-26 20:22 / 21:42 / 23:35 / 23:58, each +roughly one epoch after the 18:10 job start, while their `last.pt` files are current. For +`y26_tiles` this is confirmed exactly: the checkpoint's `best_fitness` is 0.35579, which is +its epoch-1 mAP50-95 to five decimals. ### The instability: collapse tracks the warmup LR peak @@ -106,6 +130,79 @@ precision reads 0 by the 0/0 convention, not from false positives). This matters interpretation — the instability suppresses detections, so a checkpoint caught inside the dip understates recall catastrophically and would badly misrepresent the baseline. +### The ckpt slice ceiling: why the tiles arms stopped advancing (2026-07-29) + +This is a **second, independent limitation** and must not be folded into the LR instability +above. The instability says an arm's scores got worse; this says an arm could not advance at +all. From `sacct -X -D` over **158 scheduling segments** across the five jobs and 3.5 days: + +- **Longest contiguous segment ever granted: 8.24 h.** That is the ckpt slice limit — every + clean `REQUEUED` record ends at ~08:0x elapsed. +- **Segments ≥ 9.5 h: zero.** For any arm, ever. +- The partition also got far choppier from 2026-07-28T17:00: each arm then received ~19.4 h + of compute across 20–26 segments, **16–23 of them under one hour**. On 2026-07-29 alone the + five jobs accumulated **115 segments**. + +Against that ceiling, epoch time decides whether an arm can progress at all: + +| arm | epoch time | vs the 8.2 h ceiling | epochs completed in the ~28 h to 07-29 14:45 PT | +|---|---|---|---| +| `y26_pano`, `y11l_pano` | ~2.6–2.8 h | fits 2–3× per slice | +4, +3 | +| `y11x_pano` | ~3.5–4 h | fits | +2 | +| `y11l_tiles`, `y26_tiles` | 5.2 h → **~9.5 h** | **exceeds** | **0, 0** | + +**Ultralytics checkpoints only at epoch boundaries.** Once an epoch takes longer than the +slice, every preemption discards the entire partial epoch and the arm can never advance — it +burns GPU-hours and logs nothing. `y11x_tiles` was dropped on 07-27 for exactly this (~10 h +epoch); the other two tiles arms are the same failure one notch slower, and it only became +visible when cluster I/O slowed down. + +**The cause is I/O, not compute**, and it is in the training logs verbatim: + +``` +WARNING ⚠️ train: Slow image access detected (ping: 0.6±0.2 ms, read: 2.1±1.3 MB/s, size: 338.1 KB). +``` + +…across 557,413 train tiles. So tiles epoch time is **node- and contention-dependent, not a +fixed property of the config**: ~5.2 h on a good ckpt landing (epochs 1–3, 07-26/27), ~9.5 h +on a contended one (measured 07-29: 54% of an epoch in 5.15 h), and **~4.1 h on a dedicated +`gpu-l40s` L40S** with no allocation-mates competing for the filesystem. Note this also means +a faster GPU alone would not help; and `gpu-l40s` nodes report `TMP_DISK=0`, so the dataset +cannot be staged to node-local disk to fix it. + +**Consequence for the record.** Left on ckpt, the three pano arms plausibly reach the +60-epoch budget or their `patience` stop, while the two tiles arms stay at epoch 1 +indefinitely. That is a **structural gap, not a slow one**, and reporting it as "undertrained" +would misdescribe it. + +#### The `y26_tiles_l40s` fork (job `37889646`) + +To answer *"does the tiles representation recover like pano did?"* without cancelling +anything, `y26_tiles` was forked onto **one** of makelab's two `gpu-l40s` GPUs +(`--time=72:00:00`, node `g3100`), leaving the other free for students. `y26_tiles` was chosen +over `y11l_tiles` because YOLO26 is the architecture that demonstrably recovers (`y26_pano` +round-tripped) and it sat at 0.280 vs 0.042 — the likelier of the two to yield a usable +number. **The original `y26_tiles` (job `37745360`) was not cancelled and still runs on ckpt.** + +Provenance, because a fork is easy to get silently wrong: + +- Resumed from the sha256-verified durable snapshot's `last.pt` (see "Where the weights live"). +- **Only these keys were rewritten** inside the checkpoint's `train_args`: `project`, `name`, + `save_dir`, `model`, `resume`. **This rewrite is mandatory.** Ultralytics' `resume=True` + restores *every* arg from the checkpoint — including `save_dir` — so an un-rewritten fork + resumes into the **original** run directory and corrupts a live training arm. +- Asserted unchanged on read-back: `epoch=2`, `best_fitness=0.35579`, `epochs=60`, `lr0`, + `lrf`, `warmup_epochs=3.0`, `batch=6`, `imgsz=1024`, `seed=0`, `patience=20`, `data`. So the + fork is a faithful continuation of the same LR schedule, with optimizer and EMA state + intact — Ultralytics confirms `Resuming training … from epoch 4 to 60 total epochs`. +- **Two lineages now share epochs 1–3.** Any reported tiles number must say which lineage it + came from. In practice they may never diverge, since the ckpt original has completed no + epoch since 07-28. +- **What this does and does not buy.** Per-epoch time is unchanged by the move (same shared + filesystem); what changes is the uninterrupted window. At ~4.1 h/epoch, 72 h yields roughly + 17 epochs — enough to show whether tiles turns the corner (`y26_pano` turned by ep9), but + **not** a converged 60-epoch tiles run. + ### What is reportable today Epoch 1–2 are **not** a "pretrained artifact" — an earlier version of this record said so @@ -117,6 +214,20 @@ The honest caveat: these checkpoints are **undertrained** relative to a stable s any benchmark number from them is a **lower bound** on supervised-YOLO performance, and must be reported as such. +**As of 2026-07-29 that caveat is per-arm, not uniform** — an earlier version of this record +applied it to the whole grid, which now understates one arm and *overstates* the others: + +| arm | what `best.pt` actually is | how to report it | +|---|---|---| +| `y26_pano` | epoch 18 of 60, still improving | undertrained relative to a 60-epoch stable schedule, but a genuinely trained model. A normal "lower bound" caveat. | +| `y11l_pano`, `y11x_pano`, `y11l_tiles`, `y26_tiles` | **epoch 1 of 60** | "lower bound" is too soft. These are *one-epoch models*, and the write-up must say so in those words — a reader will otherwise assume `best.pt` came from a converged run. | + +The distinction matters in both directions. Discounting `y26_pano` as "a 1-epoch model" would +understate the supervised baseline; reporting the four frozen arms as merely "undertrained" +would overstate it. Whether `y11l_pano` clears its ep1 fitness before its wall (it is ~1–2 +epochs away) decides which row it belongs in, so **re-check `best.pt`'s mtime at eval time** +rather than trusting this table. + ## Pre-registered evaluation & checkpoint-selection protocol (issue #71) Fixed in writing 2026-07-28, **before any benchmark evaluation of any YOLO @@ -197,7 +308,11 @@ protocol governs the results that will replace that note. earlier draft of this record said `fixed:0.03` — that was the runbook's old default, not what ran.) - **Toolchain:** Ultralytics 8.4.105 · Python 3.11.15 · torch 2.13.0+cu126. -- **Hardware:** NVIDIA L40 (45 GB), 1 GPU/job, Hyak `ckpt-g2` (preemptable/requeue). +- **Hardware:** NVIDIA L40 (45 GB), 1 GPU/job, Hyak `ckpt-g2` (preemptable/requeue). The + `y26_tiles_l40s` fork instead runs on Hyak `gpu-l40s` (makelab's own allocation, + non-preemptable, `MaxTime=UNLIMITED`), 1 GPU, NVIDIA L40S (45 GB), node `g3100`. Because + ckpt schedules onto whatever GPU is idle (l40/l40s/h200), per-epoch wall time varies across + requeues even within one arm — see "The ckpt slice ceiling". - **Hyperparameters (resolved):** `epochs=60`, `patience=20`, `optimizer=auto`, `lr0=0.01`, `lrf=0.01`, `momentum=0.937`, `weight_decay=0.0005`, `warmup_epochs=3.0`, `close_mosaic=10`, `amp=true`, `seed=0`. Per-run detail in each `runs//args.yaml`. @@ -209,14 +324,41 @@ protocol governs the results that will replace that note. batch-12 resubmit 37809205, scancel'd 2026-07-27) · y26_tiles 37745360 · y11l_pano 37745361 · y11x_pano 37745362 · y26_pano 37745363. The job↔config map was read off the `yolo_train_.out` log headers during the 2026-07-27 preemption check. - Dataset-download job 37649635; prep job 37677130. -- **Run dates:** 2026-07-26 → in progress as of 2026-07-28. + Dataset-download job 37649635; prep job 37677130. The `y26_tiles_l40s` fork is job + **37889646** (`gpu-l40s`, submitted 2026-07-29 ~13:55 PT). +- **Run dates:** 2026-07-26 → in progress as of 2026-07-29. +- **Mail flags are not in the launcher.** The five original jobs carry + `MailType=END,FAIL,TIME_LIMIT`, which was passed on the 07-26 sbatch command line, not set by + `run_yolo_train.slurm` (a fresh submission of the same script gets no mail config). On a + preemptable partition that means one email per preemption — 115 of them on 2026-07-29 alone. + Worth omitting, or narrowing to `TIME_LIMIT`, for the #70 rerun. ## Where the weights live `best.pt` files are **not** in git. Durable homes: -_TODO — stage to Hugging Face (`projectsidewalk/…`) or lab storage and record the URL here._ +**Staged to durable lab storage on 2026-07-29, while the runs were still going:** + +``` +/gscratch/makelab/jonf/rampnet_yolo_baseline_51/ + /{best.pt,last.pt,results.csv,args.yaml} # 5 arms, 1.8 GB total + MANIFEST.md # Slurm job IDs, per-arm epochs/metrics, sha256 of every weight file +``` + +Arms: `y11l_pano`, `y11x_pano`, `y26_pano`, `y11l_tiles`, `y26_tiles`. All **10 weight files +were sha256-verified against the source** at copy time (10 ok, 0 mismatched), and the hashes +are recorded in `MANIFEST.md` so a later copy can be checked against them. + +This was done because the weights existed **only** on `/gscratch/scrubbed`, which auto-purges +after ~21 idle days — and the run directories date from 07-26, so the clock starts the moment +the jobs stop writing. `/gscratch/makelab` is not scrubbed. + +`last.pt` is kept alongside `best.pt` deliberately: `best.pt` is what the #71/#80 protocol +reports, but only `last.pt` carries the optimizer/EMA state needed to *resume* an arm (which is +exactly what the `y26_tiles_l40s` fork above needed). + +Still open: publishing to Hugging Face (`projectsidewalk/…`) if these become a released +artifact rather than an internal baseline. Lab storage is durable but not public. Keep **every** run's `best.pt`, including the ones that collapsed. An earlier version of this record called those non-reportable epoch-1 artifacts and proposed letting them expire; From eddd34f4c81653f25e658b23e7aad56d4312ddd6 Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Wed, 29 Jul 2026 15:37:08 -0700 Subject: [PATCH 2/8] Record that the preemption-mail volume is accepted, not a defect (#51) The provenance note said the mail flags were "worth omitting" for the #70 rerun. Jon's call is the opposite -- the emails are fine and easy to delete -- so the record now says the volume is deliberate and explains what it is useful for (reading the run history), while keeping the actionable detail: the flags come from the sbatch command line, not the launcher, so a quieter rerun has to set them per job with scontrol update. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/model_comparison/yolo_baseline/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md index b06e00d..fad0faf 100644 --- a/scripts/model_comparison/yolo_baseline/README.md +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -331,7 +331,10 @@ protocol governs the results that will replace that note. `MailType=END,FAIL,TIME_LIMIT`, which was passed on the 07-26 sbatch command line, not set by `run_yolo_train.slurm` (a fresh submission of the same script gets no mail config). On a preemptable partition that means one email per preemption — 115 of them on 2026-07-29 alone. - Worth omitting, or narrowing to `TIME_LIMIT`, for the #70 rerun. + **This volume is accepted deliberately** (Jon, 2026-07-29: "I'm fine with still getting the + mail"), so it is recorded here as context for reading the run history, not as a defect to fix. + A future rerun that wants quiet should omit the flags or narrow them to `TIME_LIMIT`; note they + must be set per-job with `scontrol update`, since the launcher never sets them. ## Where the weights live From d7cf8f0bad7288b2512211cd6e23faa0a69071dc Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Wed, 29 Jul 2026 15:37:53 -0700 Subject: [PATCH 3/8] Mark the L40S epoch time as projected, not measured (#51) The 4.1 h/epoch figure for the fork was written as if measured. It is an extrapolation from 6.2 it/s at 12,123 of 92,903 iterations -- 13% of the fork'"'"'s first epoch -- and no epoch has completed on that partition yet. In a record whose whole point is that numbers be re-derivable, a projection stated as a measurement is the wrong kind of claim, and this one is load-bearing: it is the basis for "~17 epochs in 72 h." Says where the real number will come from (the time column of y26_tiles_l40s/results.csv) so it gets replaced rather than inherited. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/model_comparison/yolo_baseline/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md index fad0faf..9d96ec1 100644 --- a/scripts/model_comparison/yolo_baseline/README.md +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -166,7 +166,10 @@ WARNING ⚠️ train: Slow image access detected (ping: 0.6±0.2 ms, read: 2.1± …across 557,413 train tiles. So tiles epoch time is **node- and contention-dependent, not a fixed property of the config**: ~5.2 h on a good ckpt landing (epochs 1–3, 07-26/27), ~9.5 h on a contended one (measured 07-29: 54% of an epoch in 5.15 h), and **~4.1 h on a dedicated -`gpu-l40s` L40S** with no allocation-mates competing for the filesystem. Note this also means +`gpu-l40s` L40S** with no allocation-mates competing for the filesystem. That last figure is +**projected, not yet measured** — extrapolated from 6.2 it/s at 12,123/92,903 iterations (13% of +the fork's first epoch); replace it with the `time` column of `y26_tiles_l40s/results.csv` once an +epoch completes. Note this also means a faster GPU alone would not help; and `gpu-l40s` nodes report `TMP_DISK=0`, so the dataset cannot be staged to node-local disk to fix it. From 7afa60c8c6c13f23ebcd19b0cd16b3f5a0ceea4f Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Wed, 29 Jul 2026 20:49:31 -0700 Subject: [PATCH 4/8] Update the record to the 20:45 PT check: two arms cleared ep1, epoch time measured (#51) Three substantive changes, all from live data six hours after the previous check. y11l_pano unfroze its best.pt at ep14 (mAP50-95 0.437 > ep1 0.423 after 12 epochs unbeaten), and y26_pano advanced to ep20. So two of five arms now hold genuinely trained checkpoints and only three are one-epoch models. The per-arm caveat table is re-sorted accordingly, and now says outright that it has already moved once so the split must be re-derived at eval time rather than inherited. The fork'"'"'s epoch time is measured rather than projected: 4.94 h from the time column of its completed ep4, against the 4.1 h I extrapolated from 13% of that epoch -- about 20% low, the usual direction. Its ep5 is tracking ~4.0 h, so the dedicated-node figure is a ~4-5 h range and the 72 h budget buys ~14 epochs rather than ~17. The fork'"'"'s first new epoch also landed: ep4 is slightly below ep3, which is where the LR schedule puts it, so the recovery question stays open until ~ep9. Refines the ceiling finding in a way that matters for planning: y11x_pano has now also stopped advancing -- no epoch in ~15 h -- despite a ~3.4 h epoch that fits the 8.24 h nominal cap with room to spare. Its segments that evening were 0:58, 2:30, 0:09, 0:18, 0:02, 0:27. So the constraint is the effective preemption interval, not the slice cap, and "epoch < 8.24 h is safe" is the wrong rule. The arm is not wedged; its counter is live at ep10. Co-Authored-By: Claude Opus 5 (1M context) --- .../model_comparison/yolo_baseline/README.md | 88 ++++++++++++------- 1 file changed, 56 insertions(+), 32 deletions(-) diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md index 9d96ec1..5911da7 100644 --- a/scripts/model_comparison/yolo_baseline/README.md +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -58,7 +58,7 @@ There is a seventh **run directory** on scratch, `y26_tiles_l40s`, but it is **n config**: it is `y26_tiles` itself, resumed from its own epoch-3 checkpoint on a different partition with every hyperparameter unchanged. See "The `y26_tiles_l40s` fork" below. -## Status & findings (live check: 2026-07-29 ~14:45 PT, training in progress) +## Status & findings (live check: 2026-07-29 ~20:45 PT, training in progress) Val-split proxy mAP50 from `results.csv`. **Every config peaks at epoch 1, collapses at epoch 3, and recovers as the learning rate decays.** The instability is universal, not @@ -73,21 +73,28 @@ per-config — see the figures. | Config | Epochs | best-val (ep) | Latest mAP50 | `best.pt` holds | Assessment | |------------------|--------|---------------|--------------|-----------------|------------| -| `y26_pano` | 18 | 0.738 (1) | **0.717 ↑** | **ep18 — current** | ✅ round-tripped 0.738 → 0.125 @ep6 → 0.717 @ep18; the only arm whose `best.pt` has cleared ep1 | -| `y11l_pano` | 13 | 0.779 (1) | **0.727 ↑** | ep1 | 🟡 recovering hard — 0.000 @ep4–5 → 0.183 @ep10 → 0.727 @ep13; ~1–2 epochs from unfreezing `best.pt` | -| `y11x_pano` | 9 | 0.777 (1) | 0.061 ↑ | ep1 | ❌ five epochs at literal 0; `patience=20` retires it ~ep22, almost certainly still holding ep1 | +| `y26_pano` | 20 | 0.738 (1) | **0.730 ↑** | **ep20 — current** | ✅ round-tripped 0.738 → 0.125 @ep6 → 0.730 @ep20; mAP50-95 0.475 ≫ ep1's 0.413 | +| `y11l_pano` | 14 | 0.779 (1) | **0.767 ↑** | **ep14 — current** | ✅ **unfroze at ep14** — 0.000 @ep4–5 → 0.183 @ep10 → 0.767 @ep14; mAP50-95 0.437 > ep1's 0.423 | +| `y11x_pano` | 9 | 0.777 (1) | 0.061 ↑ | ep1 | ❌ five epochs at literal 0, **and now starved** — no completed epoch in ~15 h (see below) | | `y26_tiles` | 3 | 0.647 (1) | 0.280 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28; forked to `gpu-l40s` (below) | | `y11l_tiles` | 3 | 0.655 (1) | 0.042 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28 | -| `y26_tiles_l40s` | 3 (+) | inherited | — | inherited ep1 | 🆕 fork of `y26_tiles` resumed at ep4 on `gpu-l40s`, ~4.1 h/epoch (see "The ckpt slice ceiling") | +| `y26_tiles_l40s` | 4 | 0.647 (1) | 0.268 ↓ | inherited ep1 | 🆕 fork; **first new epoch landed** (ep4, 4.94 h) — still in the dip, as the LR schedule predicts | | `y11x_tiles` | 0 | — | — | — | ⏹ dropped 2026-07-27 (GPU-saturated: epoch ~10 h > ckpt slice) | -**`y26_pano`'s `best.pt` is epoch 18, not epoch 1.** Its `best.pt` and `last.pt` are -byte-identical (both sha256 `aac93e74…50bf`), which is direct evidence rather than an -inference from mtimes: mAP50-95 0.460 @ep18 vs 0.413 @ep1. The other four arms' `best.pt` -files are frozen at **epoch 1** — mtimes 2026-07-26 20:22 / 21:42 / 23:35 / 23:58, each -roughly one epoch after the 18:10 job start, while their `last.pt` files are current. For -`y26_tiles` this is confirmed exactly: the checkpoint's `best_fitness` is 0.35579, which is -its epoch-1 mAP50-95 to five decimals. +**Two arms have now cleared epoch 1, so the grid is genuinely split.** For `y26_pano` the +evidence is direct rather than an mtime inference: its `best.pt` and `last.pt` were +byte-identical at ep18 (both sha256 `aac93e74…50bf`), and it has since advanced to ep20 +(mAP50-95 0.475 vs 0.413 @ep1). **`y11l_pano` unfroze at ep14** on 2026-07-30 02:29Z — +`best.pt` mtime now equals `results.csv` mtime, and mAP50-95 0.437 finally beat ep1's 0.423 +after 12 epochs unbeaten. This was predicted one check earlier and is worth noting as +confirmation that the recovery is real, not noise. + +The remaining **three** arms' `best.pt` files are still frozen at **epoch 1** — `y11x_pano`, +`y11l_tiles`, `y26_tiles`, mtimes 2026-07-26 21:42 / 23:35 / 23:58, each roughly one epoch +after the 18:10 job start, while their `last.pt` files are current. For `y26_tiles` this is +confirmed exactly: the checkpoint's `best_fitness` is 0.35579, which is its epoch-1 mAP50-95 +to five decimals. **This split moves — re-check mtimes at eval time rather than trusting this +table.** ### The instability: collapse tracks the warmup LR peak @@ -165,18 +172,28 @@ WARNING ⚠️ train: Slow image access detected (ping: 0.6±0.2 ms, read: 2.1± …across 557,413 train tiles. So tiles epoch time is **node- and contention-dependent, not a fixed property of the config**: ~5.2 h on a good ckpt landing (epochs 1–3, 07-26/27), ~9.5 h -on a contended one (measured 07-29: 54% of an epoch in 5.15 h), and **~4.1 h on a dedicated -`gpu-l40s` L40S** with no allocation-mates competing for the filesystem. That last figure is -**projected, not yet measured** — extrapolated from 6.2 it/s at 12,123/92,903 iterations (13% of -the fork's first epoch); replace it with the `time` column of `y26_tiles_l40s/results.csv` once an -epoch completes. Note this also means +on a contended one (measured 07-29: 54% of an epoch in 5.15 h), and **4.94 h on a dedicated +`gpu-l40s` L40S** with no allocation-mates competing for the filesystem — now **measured**, from +the `time` column of the fork's completed epoch 4 (17,774.6 s). Its epoch 5 is tracking a little +faster (~4.0 h at 6.4 it/s), so treat **~4–5 h** as the dedicated-node range rather than a single +figure. An earlier draft of this file projected 4.1 h from 13% of an in-flight epoch; the real +value came in ~20% higher, which is the usual direction for an early-epoch extrapolation. Note this also means a faster GPU alone would not help; and `gpu-l40s` nodes report `TMP_DISK=0`, so the dataset cannot be staged to node-local disk to fix it. -**Consequence for the record.** Left on ckpt, the three pano arms plausibly reach the -60-epoch budget or their `patience` stop, while the two tiles arms stay at epoch 1 -indefinitely. That is a **structural gap, not a slow one**, and reporting it as "undertrained" -would misdescribe it. +**Refinement (2026-07-29 20:45 PT): the binding constraint is the *effective* preemption +interval, not the 8.24 h nominal cap.** `y11x_pano` has now also stopped advancing — no completed +epoch in ~15 h — despite an epoch of only ~3.4 h, which fits the nominal ceiling with room to +spare. Its segments that evening ran 0:58, 2:30, 0:09, 0:18, 0:02, 0:27. So the rule is not +"epoch < 8.24 h is safe"; it is **epoch < whatever contiguous run the partition currently +hands out**, and that collapsed to minutes. The arm is *not* wedged — its progress counter is +live (ep10, 12%, 7.2 it/s) — it simply keeps losing partial epochs. Any future capacity plan on +`ckpt` should budget against the *observed* segment distribution, not the slice cap. + +**Consequence for the record.** Left on ckpt, `y26_pano` and `y11l_pano` (epochs ~2.6–2.8 h) +plausibly reach the 60-epoch budget or their `patience` stop, while the two tiles arms — and now +`y11x_pano` — stay frozen at their current `best.pt`. That is a **structural gap, not a slow +one**, and reporting it as "undertrained" would misdescribe it. #### The `y26_tiles_l40s` fork (job `37889646`) @@ -202,9 +219,14 @@ Provenance, because a fork is easy to get silently wrong: came from. In practice they may never diverge, since the ckpt original has completed no epoch since 07-28. - **What this does and does not buy.** Per-epoch time is unchanged by the move (same shared - filesystem); what changes is the uninterrupted window. At ~4.1 h/epoch, 72 h yields roughly - 17 epochs — enough to show whether tiles turns the corner (`y26_pano` turned by ep9), but - **not** a converged 60-epoch tiles run. + filesystem); what changes is the uninterrupted window. At the measured ~4–5 h/epoch, 72 h yields + roughly **14 epochs**, reaching ~ep18 — enough to show whether tiles turns the corner + (`y26_pano` turned by ep9), but **not** a converged 60-epoch tiles run. +- **First fork epoch is in: ep4 = mAP50 0.268 / mAP50-95 0.111**, marginally *below* ep3's + 0.280 / 0.119, with precision 0.497 and recall 0.201. This is the expected place in the + schedule, not a bad sign: ep3 is the warmup LR peak and the pano arms bottomed at ep4–ep6 + before turning. The same "stops firing" signature (recall collapses, precision holds) is + present. **The recovery question stays open until ~ep9–10**, i.e. ~20–25 h of fork time. ### What is reportable today @@ -222,14 +244,16 @@ applied it to the whole grid, which now understates one arm and *overstates* the | arm | what `best.pt` actually is | how to report it | |---|---|---| -| `y26_pano` | epoch 18 of 60, still improving | undertrained relative to a 60-epoch stable schedule, but a genuinely trained model. A normal "lower bound" caveat. | -| `y11l_pano`, `y11x_pano`, `y11l_tiles`, `y26_tiles` | **epoch 1 of 60** | "lower bound" is too soft. These are *one-epoch models*, and the write-up must say so in those words — a reader will otherwise assume `best.pt` came from a converged run. | - -The distinction matters in both directions. Discounting `y26_pano` as "a 1-epoch model" would -understate the supervised baseline; reporting the four frozen arms as merely "undertrained" -would overstate it. Whether `y11l_pano` clears its ep1 fitness before its wall (it is ~1–2 -epochs away) decides which row it belongs in, so **re-check `best.pt`'s mtime at eval time** -rather than trusting this table. +| `y26_pano` (ep20), `y11l_pano` (ep14) | genuinely trained, still improving | undertrained relative to a 60-epoch stable schedule, but real models. A normal "lower bound" caveat. | +| `y11x_pano`, `y11l_tiles`, `y26_tiles` | **epoch 1 of 60** | "lower bound" is too soft. These are *one-epoch models*, and the write-up must say so in those words — a reader will otherwise assume `best.pt` came from a converged run. | + +The distinction matters in both directions. Discounting `y26_pano` or `y11l_pano` as "1-epoch +models" would understate the supervised baseline; reporting the three frozen arms as merely +"undertrained" would overstate it. + +**This table has already moved once** — `y11l_pano` was in the bottom row at the 14:45 PT check +and moved up by 20:45 PT, six hours later. So **re-check each `best.pt`'s mtime against its +`results.csv` at eval time** and place the arms then; do not inherit this split. ## Pre-registered evaluation & checkpoint-selection protocol (issue #71) From ac0b4dc2c437c6fbc31f1435ee6a1e47f4b7db08 Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Fri, 31 Jul 2026 12:06:27 -0700 Subject: [PATCH 5/8] Say how to move one arm off ckpt, since the account alone is not enough (#51) The train-stage caveat told the reader to run y11x_tiles "only on a non-preemptable partition" without saying how, and the only account-ish knob sub() exposes is -A. But run_yolo_train.slurm hardcodes "#SBATCH -p ckpt-g2 / -q ckpt-gpu", so an account-only override leaves the job on ckpt and faithfully reproduces the 8.24 h slice ceiling the move exists to escape -- silently, as a mysterious repeat of the original failure. Spell out the full hand-submit line (-p / -A / -q / --time) and note that the launcher sets no MailType, so a hand-submitted arm is silent; the 2026-07-26 arms' mail came from their sbatch CLI, not the script. Comment-only; no change to what the six-arm train stage submits. Verified against the live fork (job 37889646: Partition=gpu-l40s, Account= gpu-l40s-makelab, QOS=normal), which is the only arm submitted this way. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) --- hyak_yolo_runbook.sh | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/hyak_yolo_runbook.sh b/hyak_yolo_runbook.sh index 80a14ec..1784674 100644 --- a/hyak_yolo_runbook.sh +++ b/hyak_yolo_runbook.sh @@ -166,6 +166,17 @@ train) # Caveats from the 2026-07 runs: y11x_tiles never finished an epoch inside a ckpt # scheduling slice (dropped 2026-07-27, even after a 3->12 batch bump) — run it only # on a non-preemptable partition; YOLO11-pano collapsed at physical batch 2-4 (#70). + # + # Doing that means overriding the PARTITION and QOS, not just the account — + # run_yolo_train.slurm hardcodes "#SBATCH -p ckpt-g2 / -q ckpt-gpu", so an + # account-only override leaves the job on ckpt and faithfully reproduces the very + # 8.24 h slice ceiling you are trying to escape (#51). Submit such an arm by hand: + # PYTHON=$PYBIN YOLO_CKPT=yolo11x.pt YOLO_DATA=$YOLODATA/tiles/data.yaml \ + # YOLO_IMGSZ=1024 BATCH=6 NAME=y11x_tiles \ + # sbatch -p gpu-l40s -A gpu-l40s-makelab -q normal --time=72:00:00 $SLURM + # The launcher sets no MailType either, so a hand-submitted arm is silent — the + # 2026-07-26 arms' mail came from their sbatch CLI. Add it after submit with: + # scontrol update JobId= MailType=END,FAIL,TIME_LIMIT MailUser= sub yolo11l.pt "$YOLODATA/tiles/data.yaml" 1024 6 y11l_tiles sub yolo11x.pt "$YOLODATA/tiles/data.yaml" 1024 12 y11x_tiles sub "$YOLO26" "$YOLODATA/tiles/data.yaml" 1024 6 y26_tiles From 3dacca62996cc2bc6cc7e20d4cf83686449acc5a Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Tue, 4 Aug 2026 11:05:18 -0700 Subject: [PATCH 6/8] Review fixes for #87: 18-vs-20 remnant, the y11x_tiles restart, snapshot staleness (#51) - The status-table blockquote still said "18 epochs below" from the 14:45 PT check; the 20:45 table it introduces says 20. Stale remnant, now consistent. - Record the y11x_tiles restart (fresh, gpu-l40s, job 38063498, batch 6) in the grid section and its status row - the 2026-07-31 decision comment in #51 explicitly earmarked this status change for this file once the arm submitted, which happened while the PR was open. - Blockquote pointer to the other post-snapshot developments (fork TIMEOUT at ep18 on 08-01, y11x_pano to Tillicum on 08-04) so the merged snapshot says where its own successors live instead of implying it is current. - "Where the weights live": the durable pack is a point-in-time copy, not a mirror - per #51's 08-02 check it predates every checkpoint now worth keeping, and the completed fork's weights exist only on scrubbed with the idle-purge clock running (~08-22). - Fork provenance: bridge the epoch=2 (0-based) / "epoch-3 checkpoint" / "epoch 4" numbering so the three numbers read as one fact; and mark the shared-lineage caveat as now binding (ckpt original reached ep7 on 08-02). - Runbook: say why the hand-submit example uses BATCH=6 when the grid line above it says 12 (the #51 decision rationale, condensed). Co-Authored-By: Claude Fable 5 --- hyak_yolo_runbook.sh | 3 ++ .../model_comparison/yolo_baseline/README.md | 49 +++++++++++++++---- 2 files changed, 42 insertions(+), 10 deletions(-) diff --git a/hyak_yolo_runbook.sh b/hyak_yolo_runbook.sh index 1784674..e06bce6 100644 --- a/hyak_yolo_runbook.sh +++ b/hyak_yolo_runbook.sh @@ -174,6 +174,9 @@ train) # PYTHON=$PYBIN YOLO_CKPT=yolo11x.pt YOLO_DATA=$YOLODATA/tiles/data.yaml \ # YOLO_IMGSZ=1024 BATCH=6 NAME=y11x_tiles \ # sbatch -p gpu-l40s -A gpu-l40s-makelab -q normal --time=72:00:00 $SLURM + # (BATCH=6, not this grid's 12, is deliberate — decided 2026-07-31 in #51: 6 matches the + # sibling tiles arms, and 3->12 had measured throughput-identical ~15.6 img/s — the x + # model is GPU-saturated at 1024 px — so 12 was only ever a slice-ceiling workaround.) # The launcher sets no MailType either, so a hand-submitted arm is silent — the # 2026-07-26 arms' mail came from their sbatch CLI. Add it after submit with: # scontrol update JobId= MailType=END,FAIL,TIME_LIMIT MailUser= diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md index 5911da7..ef59dcc 100644 --- a/scripts/model_comparison/yolo_baseline/README.md +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -8,8 +8,14 @@ trained on the RampNet dataset on Hyak (klone). It exists so the experiment surv > finding is summarized in `docs/model_comparison.md` — though that summary predates the > 2026-07-29 findings below (the ckpt slice ceiling, the per-arm `best.pt` split, the fork) > and is deliberately left to be updated together with the final curves. **This file is the -> current record; `docs/model_comparison.md` is behind it by design, not by oversight.** The -> training code lives at its +> current record; `docs/model_comparison.md` is behind it by design, not by oversight.** +> Post-snapshot developments (status 2026-08-04) live in #51 rather than being re-snapshotted +> here: the `y26_tiles_l40s` fork ran to its wall (TIMEOUT 2026-08-01 at ep18 — the 72 h cap +> was self-imposed; `gpu-l40s` itself has `MaxTime=UNLIMITED`), and `y11x_pano` moved to +> Tillicum on 2026-08-04 (job `207774`) to finish its schedule; both fold in with the +> final-curves refresh. The one exception, folded in below because the 2026-07-31 decision in +> #51 earmarked it for this file's grid: **`y11x_tiles` has been restarted** (see "Config +> grid"). The training code lives at its > canonical paths — `scripts/model_comparison/run_yolo_train.slurm` and the repo-root > `hyak_yolo_runbook.sh` — merged in PR #76. (This record briefly carried as-run > snapshots of both; the only drift from the canonical copies was a display-only @@ -39,7 +45,8 @@ in git, bulk/binary/regenerable out): ## Config grid -Six configs = {YOLO11l, YOLO11x, YOLO26l} × {tiles @1024, pano @1280}. One dropped. +Six configs = {YOLO11l, YOLO11x, YOLO26l} × {tiles @1024, pano @1280}. One dropped, since +restarted (see below). | Config | Model | Input | Base ckpt | Batch | |--------------|----------|--------------|---------------|-------| @@ -54,6 +61,15 @@ Batches are the as-run values from each run's committed `args.yaml`. `y11x_tiles submitted at batch 3, then resubmitted at batch 12 trying to finish an epoch inside the ckpt scheduling slice (its `args.yaml` is the batch-12 attempt); neither completed one. +**Update 2026-08-04: `y11x_tiles` is training after all** — restarted **fresh** on the +non-preemptable `gpu-l40s` partition (job `38063498`, running as of 2026-08-04), per the +decision recorded 2026-07-31 in #51. Batch is **6**, matching the sibling tiles arms: the +3→12 history had measured throughput batch-independent (~15.6 img/s — the x model is +GPU-saturated at 1024 px), so 12 was only ever a slice-ceiling workaround with no reason to +survive off ckpt. `optimizer=auto` is kept, so the arm fills the grid hole rather than +creating a seventh config. Its run directory held no `last.pt` (verified empty before +submission), so unlike the fork below it is a clean lineage with no shared-epochs caveat. + There is a seventh **run directory** on scratch, `y26_tiles_l40s`, but it is **not a seventh config**: it is `y26_tiles` itself, resumed from its own epoch-3 checkpoint on a different partition with every hyperparameter unchanged. See "The `y26_tiles_l40s` fork" below. @@ -66,7 +82,7 @@ per-config — see the figures. > **The numbers in this table are ahead of the committed `runs/*/results.csv` and > `figures/*.png`**, which are still the 2026-07-28 snapshot (e.g. `y26_pano` has 14 rows -> there vs 18 epochs below). The CSVs and figures will be re-pulled and regenerated once +> there vs 20 epochs below). The CSVs and figures will be re-pulled and regenerated once > the arms stop, so the record ends up internally consistent against *final* curves rather > than a moving target. Until then, prefer this table for status and the CSVs for the > per-epoch shape of the collapse. @@ -79,7 +95,7 @@ per-config — see the figures. | `y26_tiles` | 3 | 0.647 (1) | 0.280 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28; forked to `gpu-l40s` (below) | | `y11l_tiles` | 3 | 0.655 (1) | 0.042 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28 | | `y26_tiles_l40s` | 4 | 0.647 (1) | 0.268 ↓ | inherited ep1 | 🆕 fork; **first new epoch landed** (ep4, 4.94 h) — still in the dip, as the LR schedule predicts | -| `y11x_tiles` | 0 | — | — | — | ⏹ dropped 2026-07-27 (GPU-saturated: epoch ~10 h > ckpt slice) | +| `y11x_tiles` | 0 | — | — | — | ⏹ dropped 2026-07-27 (GPU-saturated: epoch ~10 h > ckpt slice); **restarted fresh on `gpu-l40s` 2026-08 — see "Config grid"** | **Two arms have now cleared epoch 1, so the grid is genuinely split.** For `y26_pano` the evidence is direct rather than an mtime inference: its `best.pt` and `last.pt` were @@ -211,13 +227,18 @@ Provenance, because a fork is easy to get silently wrong: `save_dir`, `model`, `resume`. **This rewrite is mandatory.** Ultralytics' `resume=True` restores *every* arg from the checkpoint — including `save_dir` — so an un-rewritten fork resumes into the **original** run directory and corrupts a live training arm. -- Asserted unchanged on read-back: `epoch=2`, `best_fitness=0.35579`, `epochs=60`, `lr0`, - `lrf`, `warmup_epochs=3.0`, `batch=6`, `imgsz=1024`, `seed=0`, `patience=20`, `data`. So the - fork is a faithful continuation of the same LR schedule, with optimizer and EMA state - intact — Ultralytics confirms `Resuming training … from epoch 4 to 60 total epochs`. +- Asserted unchanged on read-back: `epoch=2` (0-based last-*completed* epoch, i.e. the + epoch-3 checkpoint, which resumes as the 1-based "epoch 4" below), `best_fitness=0.35579`, + `epochs=60`, `lr0`, `lrf`, `warmup_epochs=3.0`, `batch=6`, `imgsz=1024`, `seed=0`, + `patience=20`, `data`. So the fork is a faithful continuation of the same LR schedule, with + optimizer and EMA state intact — Ultralytics confirms `Resuming training … from epoch 4 to + 60 total epochs`. - **Two lineages now share epochs 1–3.** Any reported tiles number must say which lineage it came from. In practice they may never diverge, since the ckpt original has completed no - epoch since 07-28. + epoch since 07-28. *(Update 2026-08-04: they **have** diverged — the ckpt original resumed + completing epochs, reaching ep7 on 08-02, while the fork ran to ep18, so each lineage now + has its own post-ep3 epochs and the which-lineage rule is binding, not theoretical. See + #51.)* - **What this does and does not buy.** Per-epoch time is unchanged by the move (same shared filesystem); what changes is the uninterrupted window. At the measured ~4–5 h/epoch, 72 h yields roughly **14 epochs**, reaching ~ep18 — enough to show whether tiles turns the corner @@ -383,6 +404,14 @@ This was done because the weights existed **only** on `/gscratch/scrubbed`, whic after ~21 idle days — and the run directories date from 07-26, so the clock starts the moment the jobs stop writing. `/gscratch/makelab` is not scrubbed. +**It is a point-in-time copy, not a mirror.** As of #51's 2026-08-02 check the snapshot still +holds the 07-29 state — it predates every checkpoint now worth keeping, and the completed +`y26_tiles_l40s` fork is absent from it entirely. The fork stopped writing on 2026-08-01 +(TIMEOUT at ep18), so its weights sit on `/gscratch/scrubbed` with the ~21-day idle clock +running (≈2026-08-22) and no durable copy. Re-staging has begun per-arm where an arm needed +it (`y11x_pano`, re-staged 2026-08-03 for its Tillicum migration); the full re-stage belongs +with the final-curves refresh, but the fork's weights should not wait that long. + `last.pt` is kept alongside `best.pt` deliberately: `best.pt` is what the #71/#80 protocol reports, but only `last.pt` carries the optimizer/EMA state needed to *resume* an arm (which is exactly what the `y26_tiles_l40s` fork above needed). From 906b454d5e848d5527604925b3d7909b5e4d793b Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Tue, 4 Aug 2026 11:10:35 -0700 Subject: [PATCH 7/8] Correct the restart note to the as-run facts, and drop the runbook block (#51) Two corrections to the previous review-fix commit, both against the committed as-run record on exp/y11x-tiles-restart-51: - The y11x_tiles restart ran at BATCH=12 on 2026-08-03 (job 38063498, g3103, --time=7-00:00:00) - not the batch 6 the 2026-07-31 decision comment specified. The README now records the as-run value, names the deviation explicitly, and defers to args.yaml as the authority at eval time. - The runbook hand-submit comment (ac0b4dc + its review rider) is dropped, reverting hyak_yolo_runbook.sh to main: exp/y11x-tiles-restart-51 rewrites the same train-stage region with the richer as-run submission block, so keeping a second copy here guaranteed a merge conflict and preserved a batch-6/72h example that contradicts what actually ran. Its two unique nuggets survive elsewhere: the MailType note is already in this README's provenance section, and the partition-override trap is recorded in #51's 2026-07-31 decision comment. This PR is now README-only again, as its body describes. Co-Authored-By: Claude Fable 5 --- hyak_yolo_runbook.sh | 14 -------------- .../model_comparison/yolo_baseline/README.md | 19 +++++++++++-------- 2 files changed, 11 insertions(+), 22 deletions(-) diff --git a/hyak_yolo_runbook.sh b/hyak_yolo_runbook.sh index e06bce6..80a14ec 100644 --- a/hyak_yolo_runbook.sh +++ b/hyak_yolo_runbook.sh @@ -166,20 +166,6 @@ train) # Caveats from the 2026-07 runs: y11x_tiles never finished an epoch inside a ckpt # scheduling slice (dropped 2026-07-27, even after a 3->12 batch bump) — run it only # on a non-preemptable partition; YOLO11-pano collapsed at physical batch 2-4 (#70). - # - # Doing that means overriding the PARTITION and QOS, not just the account — - # run_yolo_train.slurm hardcodes "#SBATCH -p ckpt-g2 / -q ckpt-gpu", so an - # account-only override leaves the job on ckpt and faithfully reproduces the very - # 8.24 h slice ceiling you are trying to escape (#51). Submit such an arm by hand: - # PYTHON=$PYBIN YOLO_CKPT=yolo11x.pt YOLO_DATA=$YOLODATA/tiles/data.yaml \ - # YOLO_IMGSZ=1024 BATCH=6 NAME=y11x_tiles \ - # sbatch -p gpu-l40s -A gpu-l40s-makelab -q normal --time=72:00:00 $SLURM - # (BATCH=6, not this grid's 12, is deliberate — decided 2026-07-31 in #51: 6 matches the - # sibling tiles arms, and 3->12 had measured throughput-identical ~15.6 img/s — the x - # model is GPU-saturated at 1024 px — so 12 was only ever a slice-ceiling workaround.) - # The launcher sets no MailType either, so a hand-submitted arm is silent — the - # 2026-07-26 arms' mail came from their sbatch CLI. Add it after submit with: - # scontrol update JobId= MailType=END,FAIL,TIME_LIMIT MailUser= sub yolo11l.pt "$YOLODATA/tiles/data.yaml" 1024 6 y11l_tiles sub yolo11x.pt "$YOLODATA/tiles/data.yaml" 1024 12 y11x_tiles sub "$YOLO26" "$YOLODATA/tiles/data.yaml" 1024 6 y26_tiles diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md index ef59dcc..692c37b 100644 --- a/scripts/model_comparison/yolo_baseline/README.md +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -62,13 +62,16 @@ submitted at batch 3, then resubmitted at batch 12 trying to finish an epoch ins ckpt scheduling slice (its `args.yaml` is the batch-12 attempt); neither completed one. **Update 2026-08-04: `y11x_tiles` is training after all** — restarted **fresh** on the -non-preemptable `gpu-l40s` partition (job `38063498`, running as of 2026-08-04), per the -decision recorded 2026-07-31 in #51. Batch is **6**, matching the sibling tiles arms: the -3→12 history had measured throughput batch-independent (~15.6 img/s — the x model is -GPU-saturated at 1024 px), so 12 was only ever a slice-ceiling workaround with no reason to -survive off ckpt. `optimizer=auto` is kept, so the arm fills the grid hole rather than -creating a seventh config. Its run directory held no `last.pt` (verified empty before -submission), so unlike the fork below it is a clean lineage with no shared-epochs caveat. +non-preemptable `gpu-l40s` partition on 2026-08-03 (job `38063498`, node `g3103`, +`--time=7-00:00:00`, started immediately), taking the venue decided 2026-07-31 in #51 once +the `y26_tiles_l40s` fork released the lab node. As-run config: **batch 12**, with +`optimizer=auto` kept per the decision so the arm fills the grid hole rather than creating a +differently-scheduled config; the as-run submit line is committed on +`exp/y11x-tiles-restart-51`. The batch **deviates from the 2026-07-31 decision comment**, +which specified 6 to match the sibling tiles arms; the submission followed this grid's +committed batch-12 pin instead — at eval time its `args.yaml` is the authority, not either +note. It is a fresh start (no `last.pt` existed), so unlike the fork below it is a clean +lineage with no shared-epochs caveat. There is a seventh **run directory** on scratch, `y26_tiles_l40s`, but it is **not a seventh config**: it is `y26_tiles` itself, resumed from its own epoch-3 checkpoint on a different @@ -95,7 +98,7 @@ per-config — see the figures. | `y26_tiles` | 3 | 0.647 (1) | 0.280 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28; forked to `gpu-l40s` (below) | | `y11l_tiles` | 3 | 0.655 (1) | 0.042 ↓ | ep1 | ⏸ **blocked** — no completed epoch since 2026-07-28 | | `y26_tiles_l40s` | 4 | 0.647 (1) | 0.268 ↓ | inherited ep1 | 🆕 fork; **first new epoch landed** (ep4, 4.94 h) — still in the dip, as the LR schedule predicts | -| `y11x_tiles` | 0 | — | — | — | ⏹ dropped 2026-07-27 (GPU-saturated: epoch ~10 h > ckpt slice); **restarted fresh on `gpu-l40s` 2026-08 — see "Config grid"** | +| `y11x_tiles` | 0 | — | — | — | ⏹ dropped 2026-07-27 (GPU-saturated: epoch ~10 h > ckpt slice); **restarted fresh on `gpu-l40s` 2026-08-03 — see "Config grid"** | **Two arms have now cleared epoch 1, so the grid is genuinely split.** For `y26_pano` the evidence is direct rather than an mtime inference: its `best.pt` and `last.pt` were From 01e6dc54dd37e2db84ddf0417dd588eab55d8c1c Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Tue, 4 Aug 2026 11:12:47 -0700 Subject: [PATCH 8/8] The snapshot caveat was itself stale: the re-stage already happened 2026-08-03 (#51) The previous commit warned the completed fork weights existed only on scrubbed with the purge clock running. True only for 08-01 -> 08-03: the durable pack was refreshed on 2026-08-03 to all six arms (24/24 sha256-verified, superseded manifest preserved as MANIFEST-2026-07-29.md), and the 08-04 Tillicum migration already resumed from it, sha256-matched. The caveat now records that lifecycle - staleness caught, refresh done, check MANIFEST.md dates before trusting a copy - instead of re-raising a solved problem. Co-Authored-By: Claude Fable 5 --- .../model_comparison/yolo_baseline/README.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/scripts/model_comparison/yolo_baseline/README.md b/scripts/model_comparison/yolo_baseline/README.md index 692c37b..9dfe4d2 100644 --- a/scripts/model_comparison/yolo_baseline/README.md +++ b/scripts/model_comparison/yolo_baseline/README.md @@ -407,13 +407,16 @@ This was done because the weights existed **only** on `/gscratch/scrubbed`, whic after ~21 idle days — and the run directories date from 07-26, so the clock starts the moment the jobs stop writing. `/gscratch/makelab` is not scrubbed. -**It is a point-in-time copy, not a mirror.** As of #51's 2026-08-02 check the snapshot still -holds the 07-29 state — it predates every checkpoint now worth keeping, and the completed -`y26_tiles_l40s` fork is absent from it entirely. The fork stopped writing on 2026-08-01 -(TIMEOUT at ep18), so its weights sit on `/gscratch/scrubbed` with the ~21-day idle clock -running (≈2026-08-22) and no durable copy. Re-staging has begun per-arm where an arm needed -it (`y11x_pano`, re-staged 2026-08-03 for its Tillicum migration); the full re-stage belongs -with the final-curves refresh, but the fork's weights should not wait that long. +**It is a point-in-time copy that gets refreshed, not a mirror.** #51's 2026-08-02 check +found exactly the failure mode to expect: the pack still held the 07-29 state, predating +every checkpoint then worth keeping, with the completed `y26_tiles_l40s` fork — the only +converged run — absent entirely. It was refreshed on **2026-08-03** to all six arms, 24/24 +files sha256-verified, with the superseded manifest preserved as `MANIFEST-2026-07-29.md` +rather than overwritten (that one had captured `y11x_pano` mid-collapse at 9 ep / 0.024, +which reads as a result if taken at face value). The 08-04 Tillicum migration resumed +`y11x_pano` from this snapshot's `last.pt`, sha256-matched end to end (#51). So: **trust a +copy only after checking `MANIFEST.md`'s date against the live run state** — the runs keep +moving, and a stale snapshot does not announce itself. `last.pt` is kept alongside `best.pt` deliberately: `best.pt` is what the #71/#80 protocol reports, but only `last.pt` carries the optimizer/EMA state needed to *resume* an arm (which is