From 9430266334f04b2d38b327bb01dbbad269d978e5 Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Thu, 30 Jul 2026 07:07:23 -0700 Subject: [PATCH 1/7] Add docs/tillicum.md: access, cost model, and the Slurm migration diff MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The makelab group was provisioned on Tillicum 2026-07-29. Tillicum is not klone with different hostnames — no partitions (QoS only), a hard 8 CPU : 1 GPU ratio, a 24 h ceiling on the default QoS, and usage billing at $0.90/GPU-hour. Our run_yolo_train.slurm is non-submittable as written; the diff table records exactly what changes and why. Motivation is #51/#70: klone's ckpt partition consumed 496.5 GPU-hours on the YOLO baseline and, in one night, produced 170 job restarts and zero completed epochs across five of six arms. Tillicum does not preempt. Flags the two decisions that need measurement rather than argument: whether our CUDA 11.8 conda env transfers to H200/Rocky 9 (vs. an NGC container), and whether 16 dataloader CPUs makes the I/O-bound tiles epoch more than 2x faster — which is the only thing that justifies its 2x billing rate, and which also settles whether tiles-vs-pano in #51 is an architecture finding or a storage artifact. Sections that are inference rather than documented fact are marked UNVERIFIED; no job has been run on Tillicum yet. Co-Authored-By: Claude Opus 5 (1M context) --- docs/tillicum.md | 227 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 docs/tillicum.md diff --git a/docs/tillicum.md b/docs/tillicum.md new file mode 100644 index 0000000..6f59e20 --- /dev/null +++ b/docs/tillicum.md @@ -0,0 +1,227 @@ +# Tillicum — access, migration, and what it costs + +Working notes for running RampNet jobs on **Tillicum**, UW-IT's usage-billed GPU +cluster. The makelab group was provisioned on **2026-07-29**. + +This doc exists because Tillicum is **not** klone-with-different-hostnames — the +scheduler model, the walltime ceiling, the CPU:GPU ratio, and the cost model all +differ in ways that make our existing `.slurm` scripts **non-submittable as written**. +See [Migrating our Slurm scripts](#migrating-our-slurm-scripts) for the diff. + +> **Sourcing.** Everything in the tables below is from the [Tillicum +> docs](https://hyak.uw.edu/docs/systems/tillicum/scheduling-jobs) and the provisioning +> email (2026-07-29), read on 2026-07-30. Sections marked **UNVERIFIED** are our +> inference or are undocumented — we have not run a single job on Tillicum yet, so +> nothing here is confirmed by experience. Correct this file from measurement, not +> from re-reading the docs. + +## Why we care: the klone problem this solves + +The supervised-YOLO baseline (#51) ran on klone's `ckpt` scavenger partition, which is +free but **preemptable**. As of 2026-07-30 that had stopped working as a compute +strategy: + +- **496.5 GPU-hours consumed** on the baseline since 2026-07-24 (`sacct`, all arms). +- **170 job restarts in one night** across five jobs (39 / 37 / 33 / 33 / 28). +- **Zero completed epochs** on five of six arms over that same night; four of the six + arms are still "one-epoch models" holding an ep1 `best.pt`. + +The binding constraint is not the nominal 8.24 h checkpoint slice — it is the +**effective** contiguous run the partition hands out, which collapsed to minutes. Any +arm whose epoch exceeds that interval can never complete one. See #51 and #70. + +**Tillicum does not preempt.** The docs state that even the `urgent` QoS will "not +cancel or preempt jobs that are already running." That is the entire reason to pay. + +## Access + +```bash +ssh @tillicum.hyak.uw.edu +``` + +Duo 2FA, same as klone — **Claude cannot authenticate this.** The `wsl-ssh.ps1` helper +in `dotfiles` has no `tillicum` target yet; adding one (with a `ControlPersist` master, +mirroring the `klone` block in `ssh_config.wsl`) is a prerequisite for driving Tillicum +from an agent session. Until then, Tillicum is Jon-only. + +> The get-started page writes the hostname two ways — `tillicum.hyak.uw.edu` in the +> hostname field and `tillicum.hyak.edu` in the `ssh` example. The former is almost +> certainly correct; **UNVERIFIED** until someone logs in. + +## The scheduler model: QoS, not partitions + +klone is a condo: you get partitions tied to hardware your group owns or scavenges +(`-p ckpt-g2 -q ckpt-gpu`). **Tillicum has no partitions at all.** You pick a QoS, and +billing handles the rest. + +| QoS | Max time | Max GPUs/job | Concurrent | Notes | +|---|---|---|---|---| +| `normal` | **24 h** | 16 | 48 GPUs | default | +| `debug` | 1 h | 1 | 1 job | smoke tests | +| `interactive` | 8 h | 2 | 2 jobs | `salloc` work | +| `long` | **7 days** | 16 | 96 GPUs (shared) | **pre-approval required** | +| `wide` | 24 h | unlimited | 96 GPUs (shared) | **pre-approval required** | +| `urgent` | 3 days | 64 | 96 GPUs (shared) | **pre-approval required** | + +**The 24 h ceiling on `normal` is a real constraint for us.** Our tiles epochs ran +4.1–9.5 h on klone; a 60-epoch schedule does not fit in 24 h. Two options: + +1. **Request `long` QoS** (7 days) via the Tillicum Special QoS Access Request Form. + This is the clean answer and should be requested now — it gates the #70 rerun. +2. **Chain 24 h `normal` jobs**, resuming from `weights/last.pt`. Our + `run_yolo_train.slurm` already has the resume block for this, and without + preemption a resume boundary is predictable rather than random. + +Option 2 works today and needs no approval; option 1 is less operationally fiddly. +Do 1, fall back to 2. + +## Hardware + +192 × NVIDIA **H200** (141 GB HBM3e, NVLink 4.0), 8 GPUs per node, 24 × Dell XE9680, +1,536 Intel Emerald Rapids cores, 400 Gbps NDR InfiniBand, ~3 PB flash, Rocky 9. + +Per-GPU binding: **200 GB system RAM and 8 CPUs**. This ratio is not negotiable — +the docs are explicit that exceeding 8 CPUs per GPU requires requesting *additional +GPUs*. **CPU-only jobs are prohibited**; every job must request ≥1 GPU. + +> Operational consequence: our CPU-side data prep (`run_yolo_data.slurm`, +> `run_yolo_prep.slurm`) has no business on Tillicum. Keep prep on klone, train on +> Tillicum. + +## Cost model + +**$0.90 per GPU-hour**, where **GPU-hour = elapsed wall-clock × N GPUs**. Billed +monthly via ITBill to the lab's UW worktag. (**The worktag and subscription IDs are +deliberately not recorded in this repo** — it is public. Ask Jon, or see the Tillicum +provisioning email.) + +- **100 free GPU demo hours** on the new account (≈ $90). Whether these expire is + **UNVERIFIED**. +- Budget requested 2026-07-30: **$1,500/month** ≈ **1,666 GPU-hours/month**. + +At that ceiling: + +| job shape | wall-clock hours/month | +|---|---| +| 1 GPU | ~1,666 | +| 2 GPUs | ~833 | +| 8 GPUs (full node) | ~208 | + +Monitor with `hyakusage`. + +### The 2-GPU trap for I/O-bound arms + +This is the cost decision specific to our workload. Our tiles arm is **I/O-bound, not +GPU-bound** — the same epoch took 4.1–9.5 h on klone depending only on which node it +landed on. The obvious fix is more dataloader workers, but on Tillicum **more CPUs +means more GPUs**, and billing multiplies by GPU count: + +- 1 GPU → 8 dataloader CPUs → 1× billing rate +- 2 GPUs → 16 dataloader CPUs → **2× billing rate** + +So 2 GPUs is only cheaper *per epoch* if 16 CPUs makes the tiles epoch **more than +twice** as fast. That is an empirical question, and it is the single best use of the +free demo hours. See [First runs](#first-runs-what-to-measure). + +## Storage + +`/gpfs/projects/makelab` — **1 TB**, backed up daily, "purged at the end of the +project." Note this is *not* archival: `/gscratch/makelab` on klone remains the system +of record for durable weights (see `scripts/model_comparison/yolo_baseline/README.md`). +Keep the existing pattern: **train on Tillicum, `rsync` `best.pt` back to +`/gscratch/makelab`.** + +Undocumented and pending a reply from UW-IT: the quota-increase path, what defines +"end of project," and guidance for **many-small-file** access patterns. That last one +matters to us more than bandwidth — our tiles dataset is hundreds of thousands of small +JPEGs, and `du -sh` over it on klone's `/gscratch/scrubbed` exceeds a 2-minute timeout, +which is a metadata-throughput symptom rather than a size one. + +> The docs describe demo accounts as receiving "100 GB of dedicated project storage," +> while our provisioning email says 1 TB. Assume 1 TB (the email is specific to us) but +> confirm. + +## Software environment + +Tillicum's documented path is **Apptainer containers**, not conda modules — +see [UWrc/tillicum-containers](https://github.com/UWrc/tillicum-containers), which +ships a `pytorch_timm.def` example (we use `timm` for the ConvNeXt-V2 backbone) plus +`run_inference.slurm` and `array_inference.slurm` templates. Recommended image sources +are DockerHub and the NVIDIA NGC catalog. + +**Our `environment.yml` should not be assumed to transfer.** It pins linux-64 packages +against **CUDA 11.8**, and Tillicum is H200 (sm_90) on Rocky 9. CUDA 11.8 nominally +covers sm_90, but a stack built for klone's OS and driver is not a safe bet on a +different distro and a newer card. **UNVERIFIED — do not plan around either outcome +until someone tries it.** The low-risk path is an NGC PyTorch container. + +## Migrating our Slurm scripts + +Our current header (`scripts/model_comparison/run_yolo_train.slurm`) against what +Tillicum accepts: + +| klone (current) | Tillicum | why | +|---|---|---| +| `#SBATCH -p ckpt-g2` | **delete** | no partitions exist | +| `#SBATCH -q ckpt-gpu` | `#SBATCH --qos=normal` | or `long` once approved | +| `#SBATCH --requeue` | **delete** | nothing preempts | +| `#SBATCH --time=72:00:00` | `24:00:00` | 72 h needs `long` QoS | +| `#SBATCH --gpus-per-node=1` | `#SBATCH --gres=gpu:1` | also `--gpus=1` / `-G 1` | +| `#SBATCH --cpus-per-task=12` | **`8`** (1 GPU) | 12 CPUs on 1 GPU is rejected | +| `#SBATCH --mem=64G` | keep, or up to `200G` | no default; must be explicit | +| `#SBATCH --nodes=1 --ntasks=1` | unchanged | | + +**`--cpus-per-task=12` with one GPU is the blocking incompatibility** — it is not a +tuning preference, it exceeds the hard 8:1 ratio and the job will not run. + +Also drop the inherited mail settings. Our klone jobs carry +`MailType=END,FAIL,TIME_LIMIT` from a site default or the submit environment (it is +*not* in the script), which on a preemptable partition produced ~170 emails in one +night. On Tillicum there is no requeue storm to amplify it, but set `--mail-type=NONE` +explicitly so the behavior is the script's decision rather than the site's. + +## First runs: what to measure + +Spend the free demo hours on measurement, not on a production run. Three jobs, ~15 +GPU-hours total, answering the questions the #70 budget depends on: + +1. **`debug` QoS smoke test** (1 h, ~1 GPU-h) — does our environment run at all? + This is where the conda-vs-container question gets settled. +2. **One pano epoch, 1 GPU** (~1–3 h). klone L40S baseline: 2.6–2.8 h/epoch. An H200 + should be meaningfully faster, but *how much* is the number the budget needs. +3. **One tiles epoch, 1 GPU (8 CPU) vs 2 GPUs (16 CPU)** (~10 GPU-h). Settles the + 2-GPU trap above, and — more importantly — tells us whether tiles-vs-pano in the + #51 results is an **architecture finding or a storage artifact**. That question is + currently unresolved and it materially affects what the baseline means. + +Record the measured epoch times back into this file and into #70. + +## Admin + +- **Group:** `u_hyak_tillicum_makelab` (UW Groups Service). Jon is a member manager and + can add users via the Membership tab; UW-IT can grant member-manager rights to others + on request. +- **Maintenance:** second Tuesday monthly. Join the mailing list; put it on the run + calendar so long jobs are not scheduled across it. +- **Support:** `help@uw.edu` with "Tillicum" in the subject; twice-weekly office hours. +- **Budget changes:** contact UW-IT; enforcement semantics (blocks new submissions vs. + cancels running jobs) are **UNVERIFIED** and were asked about on 2026-07-30. + +## Open questions + +Sent to UW-IT 2026-07-30, unanswered at time of writing: + +1. Does budget enforcement block *new submissions* or *cancel running jobs*? +2. Is there any way to get more CPUs per job without the GPU billing multiplier? +3. Guidance for many-small-file datasets on `/gpfs`; is packing into an archive or + container format recommended? +4. Storage: quota-increase path, and what "purged at the end of the project" means. +5. Do the 100 free demo hours expire? +6. How to request `long` QoS access. + +## Related + +- #51 — supervised YOLO baseline (the runs that motivated this) +- #70 — stabilized rerun; **this is its unblocking dependency** +- `scripts/model_comparison/yolo_baseline/README.md` — preserved klone training record +- `hyak_yolo_runbook.sh`, `hyak_qwen_runbook.sh` — klone runbooks (repo root) From 315a672f879106322bc47d5a039fe59198aa91bf Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Thu, 30 Jul 2026 09:58:41 -0700 Subject: [PATCH 2/7] Record the measured dataset size: 286 GB across ~911k files The Tillicum migration plan depends on how much data has to cross from klone's /gscratch to /gpfs, and the answer changes the approach. Measured on klone: tiles 210 GB / 718,415 files, pano 76 GB / 192,938 files. 286 GB fits the 1 TB allocation comfortably, so storage quota is not a blocker. The ~911,000 files are: this is the many-small-file pathology in pure form, and it is measured rather than assumed -- du -sh over the tiles tree exceeded a 2-minute timeout three times before completing on a 550 s budget, and even ls -lU | head -201 on tiles/images/train timed out. That is metadata throughput, not bandwidth. Consequences now in the doc: transfer tar archives rather than the tree (a per-file rsync would pay ~911k round trips), run the tar inside a Slurm job rather than on a login node, stage pano first so the smoke test is unblocked while tiles moves, and time the untar -- it is our first real measurement of whether Tillicum's flash storage fixes the bottleneck that the tiles training arm is also probing. Co-Authored-By: Claude Opus 5 (1M context) --- docs/tillicum.md | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/docs/tillicum.md b/docs/tillicum.md index 6f59e20..8865dae 100644 --- a/docs/tillicum.md +++ b/docs/tillicum.md @@ -133,9 +133,35 @@ Keep the existing pattern: **train on Tillicum, `rsync` `best.pt` back to Undocumented and pending a reply from UW-IT: the quota-increase path, what defines "end of project," and guidance for **many-small-file** access patterns. That last one -matters to us more than bandwidth — our tiles dataset is hundreds of thousands of small -JPEGs, and `du -sh` over it on klone's `/gscratch/scrubbed` exceeds a 2-minute timeout, -which is a metadata-throughput symptom rather than a size one. +matters to us more than bandwidth. + +### What actually has to move (measured 2026-07-30 on klone) + +| dataset | size | train files | val files | +|---|---|---|---| +| `yolo/tiles` | **210 GB** | 557,413 | 161,002 | +| `yolo/pano` | **76 GB** | 150,063 | 42,875 | +| **total** | **286 GB** | **~911,000 files**, ~300 KB average | | + +Two conclusions: + +- **286 GB fits the 1 TB allocation** with room for checkpoints and run dirs. Storage + quota is not a blocker for the #70 rerun. +- **~911,000 files is the blocker.** This is the many-small-file pathology in its pure + form, and it is not hypothetical: `du -sh` over the tiles tree exceeded a 2-minute + timeout three separate times before completing on a 550 s budget, and even + `ls -lU | head -201` on `tiles/images/train` timed out. That is metadata throughput, + not bandwidth. + +**So transfer an archive, never the tree.** `tar` per split on klone → move a handful of +large files → untar on Tillicum. A per-file `rsync`/`scp` would pay ~911,000 round trips. +Run the `tar` inside a Slurm job, **not on a klone login node** — that is exactly the +heavy-login-process reap that kills the SSH master. Note the untar on `/gpfs` is itself +metadata-heavy, so time it: it is our first real measurement of whether Tillicum's flash +actually fixes this, which is the same question the tiles training arm is asking. + +Staging order: **`pano` first (76 GB, 193k files)** — it unblocks the smoke test and the +pano epoch-time measurement while the much larger `tiles` archive moves behind it. > The docs describe demo accounts as receiving "100 GB of dedicated project storage," > while our provisioning email says 1 TB. Assume 1 TB (the email is specific to us) but From 5237e86cf446469f82412eb050dd2c658b43286d Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Thu, 30 Jul 2026 10:07:24 -0700 Subject: [PATCH 3/7] Add the Tillicum training script and a first-login recon pass Two pieces that need no Tillicum account, so the only remaining gate is Jon's Duo login. run_yolo_train_tillicum.slurm is a port of the klone script, kept as a separate file rather than a branch inside it: nearly every scheduler directive differs, and the klone version is the preserved record of the #51 runs and should not churn. QoS instead of partitions, 24 h instead of 72, --gres=gpu:N, 8 CPUs per GPU, no --requeue, and an explicit --mail-type=NONE so the site default cannot decide for us the way klone's did. It also separates the GPU ALLOCATION from the ultralytics DEVICE, which is the non-obvious part. Tillicum fixes CPUs at 8 per GPU, so getting 16 dataloader workers for the I/O-bound tiles arm means allocating 2 GPUs. But letting ultralytics use both puts it in DDP, changing effective batch and LR dynamics and breaking comparability with the klone single-GPU runs. So we allocate 2 and pin DEVICE=0, paying for an idle GPU on purpose: it isolates dataloader throughput as the only changed variable, which is exactly the question -- does the 2x bill buy more than a 2x speedup. tillicum_recon.sh settles, in one read-only pass, every item docs/tillicum.md currently marks UNVERIFIED: the undocumented home path (wsl-ssh.ps1 is guessing /gpfs/projects/makelab), whether long QoS is already granted, whether hyakusage and apptainer are on PATH, and whether Tillicum forces job mail the way klone's lua job_submit plugin does. Co-Authored-By: Claude Opus 5 (1M context) --- .../run_yolo_train_tillicum.slurm | 134 ++++++++++++++++++ scripts/tillicum_recon.sh | 83 +++++++++++ 2 files changed, 217 insertions(+) create mode 100644 scripts/model_comparison/run_yolo_train_tillicum.slurm create mode 100644 scripts/tillicum_recon.sh diff --git a/scripts/model_comparison/run_yolo_train_tillicum.slurm b/scripts/model_comparison/run_yolo_train_tillicum.slurm new file mode 100644 index 0000000..1f873db --- /dev/null +++ b/scripts/model_comparison/run_yolo_train_tillicum.slurm @@ -0,0 +1,134 @@ +#!/bin/bash +# Train one YOLO curb-ramp detector on the RampNet dataset, on TILLICUM. +# +# Port of run_yolo_train.slurm (klone). Kept as a SEPARATE file rather than +# branching inside the klone script, because nearly every scheduler directive +# differs and the klone version is the preserved record of the #51 runs -- it +# should not churn. See docs/tillicum.md for the directive-by-directive diff. +# +# WHAT CHANGED FROM THE KLONE VERSION, AND WHY +# -p ckpt-g2 / -q ckpt-gpu -> --qos=normal Tillicum has NO partitions; QoS only. +# --requeue -> dropped Nothing preempts on Tillicum. +# --time=72:00:00 -> 24:00:00 normal QoS caps at 24 h. 7-day jobs +# need `long` QoS (pre-approval). +# --gpus-per-node=1 -> --gres=gpu:N Tillicum's documented spelling. +# --cpus-per-task=12 -> 8 x N GPUs HARD 8:1 ratio; 12-on-1 is REJECTED. +# (no mail directive) -> --mail-type=NONE Be explicit so the site default +# cannot decide for us (on klone an +# inherited END,FAIL,TIME_LIMIT produced +# ~1 email per minute under preemption). +# +# YOLO_DATA=$DATA/pano/data.yaml YOLO_IMGSZ=1280 BATCH=4 NAME=y26_pano_till \ +# sbatch scripts/model_comparison/run_yolo_train_tillicum.slurm +# +# COST. Tillicum BILLS: $0.90/GPU-hour where GPU-hour = elapsed x N GPUs. A 24 h +# 2-GPU job is 48 GPU-hours (~$43). Do not leave allocations idle. Check `hyakusage`. +#SBATCH --qos=normal +#SBATCH --job-name=yolo_curb_ramp_train +#SBATCH --time=24:00:00 +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --mail-type=NONE +#SBATCH --output=logs/yolo_train_till_%j.out +#SBATCH --error=logs/yolo_train_till_%j.err +# gres/cpus/mem are NOT hardcoded -- see the GPUS block below, which must be kept in +# step with them. Override at submit time with: sbatch --gres=gpu:2 --cpus-per-task=16 +#SBATCH --gres=gpu:1 +#SBATCH --cpus-per-task=8 +#SBATCH --mem=200G + +set -euo pipefail + +# --------------------------------------------------------------------------------- +# The 2-GPU dataloader trick, and why DEVICE is separate from the allocation. +# +# Tillicum fixes CPUs at 8 per GPU. Our tiles arm is I/O-bound (the same epoch took +# 4.1-9.5 h on klone depending only on which node it landed on), so more dataloader +# workers is the obvious lever -- but on Tillicum more CPUs means more GPUs. +# +# The naive move -- allocate 2 GPUs and let ultralytics use both -- is NOT a clean +# experiment: 2 GPUs puts ultralytics into DDP, which changes the effective batch and +# the LR dynamics, so the result is no longer comparable to the klone single-GPU runs. +# +# So: allocate 2 GPUs for their 16 CPUs, but pin DEVICE=0 so training math is +# IDENTICAL to a 1-GPU run. The second GPU sits idle and we pay for it. That is the +# point -- it isolates dataloader throughput as the only changed variable, and it +# answers whether the 2x bill buys more than a 2x speedup. Set DEVICE=0,1 only if you +# deliberately want a DDP run, and label it as a different arm. +# --------------------------------------------------------------------------------- +DEVICE="${DEVICE:-0}" + +YOLO_CKPT="${YOLO_CKPT:-yolo11l.pt}" +YOLO_DATA="${YOLO_DATA:?set YOLO_DATA to a prepared data.yaml}" +YOLO_IMGSZ="${YOLO_IMGSZ:-1024}" +EPOCHS="${EPOCHS:-60}" +BATCH="${BATCH:--1}" # pin per config to match the klone runs +PATIENCE="${PATIENCE:-20}" +# On klone this defaulted to 0 because ultralytics `time=` restarts its clock on every +# preemption. Tillicum never preempts, so `time=` is safe here -- but leave it 0 for +# any run meant to be comparable to the #51 klone results, which are epoch-based. +TRAIN_HOURS="${TRAIN_HOURS:-0}" +NAME="${NAME:-yolo_run_till}" + +# Tillicum storage. /gpfs/projects/makelab is 1 TB, backed up daily, and PURGED AT END +# OF PROJECT -- it is NOT archival. Only best.pt matters downstream: rsync it back to +# klone /gscratch/makelab, which stays the system of record. +GPFS="${GPFS:-/gpfs/projects/makelab/$USER}" +export HF_HOME="${HF_HOME:-$GPFS/hf}" +PROJECT="${PROJECT:-$GPFS/yolo_runs}" +export YOLO_CONFIG_DIR="${YOLO_CONFIG_DIR:-$GPFS/ultralytics}" +mkdir -p "$PROJECT" "$YOLO_CONFIG_DIR" "$HF_HOME" + +# PYTHON points at an interpreter with ultralytics. On Tillicum the DOCUMENTED path is +# an Apptainer container (NGC PyTorch + `pip install ultralytics`), not a conda module +# -- our environment.yml pins CUDA 11.8 against what is now a Rocky 9 / H200 host and +# should not be assumed to transfer. Set PYTHON, or set APPTAINER_IMG to run in one. +PYTHON="${PYTHON:-python}" +APPTAINER_IMG="${APPTAINER_IMG:-}" + +echo "--- YOLO train on TILLICUM (issue #51 / #70) ---" +echo "base: ${YOLO_CKPT}" +echo "data: ${YOLO_DATA}" +echo "imgsz: ${YOLO_IMGSZ} epochs: ${EPOCHS} batch: ${BATCH} patience: ${PATIENCE}" +echo "alloc: ${SLURM_GPUS_ON_NODE:-?} GPU(s), ${SLURM_CPUS_PER_TASK:-?} CPUs on ${SLURMD_NODENAME:-?}" +echo "device: ${DEVICE} (allocation and device differ on purpose -- see header)" +echo "out: ${PROJECT}/${NAME}/weights/best.pt" +nvidia-smi --query-gpu=name,memory.total --format=csv || true +echo "------------------------------------------------" + +run_train() { + "$@" - "$YOLO_CKPT" "$YOLO_DATA" "$YOLO_IMGSZ" "$EPOCHS" "$BATCH" "$PATIENCE" \ + "$PROJECT" "$NAME" "$TRAIN_HOURS" "$DEVICE" <<'PY' +import os, sys +from ultralytics import YOLO +ckpt, data, imgsz, epochs, batch, patience, project, name, hours, device = sys.argv[1:11] +last = os.path.join(project, name, "weights", "last.pt") +# Still needed on Tillicum, but for a different reason than on klone: not preemption, +# but the 24 h normal-QoS ceiling. A 60-epoch tiles schedule does not fit in one job, +# so chain jobs and resume. Unlike klone the boundary is predictable, not random. +if os.path.exists(last): + print(f"[resume] {last} exists -> resuming", flush=True) + YOLO(last).train(resume=True) +else: + bf = float(batch) + batch_arg = int(bf) if bf.is_integer() else bf + dev = [int(d) for d in device.split(",")] if "," in device else int(device) + kw = dict( + data=data, imgsz=int(imgsz), epochs=int(epochs), batch=batch_arg, + patience=int(patience), project=project, name=name, device=dev, exist_ok=True, + ) + if float(hours) > 0: + kw["time"] = float(hours) + YOLO(ckpt).train(**kw) +PY +} + +# run_train takes the interpreter as "$@", so the container case is just a longer +# command word -- the heredoc still lands on its stdin and every shell variable is +# expanded HERE, outside the container, before apptainer is ever invoked. +# --nv exposes the GPUs; bind /gpfs so the container sees data, runs and cache. +if [ -n "$APPTAINER_IMG" ]; then + run_train apptainer exec --nv --bind /gpfs:/gpfs "$APPTAINER_IMG" python +else + run_train "$PYTHON" +fi diff --git a/scripts/tillicum_recon.sh b/scripts/tillicum_recon.sh new file mode 100644 index 0000000..de640dc --- /dev/null +++ b/scripts/tillicum_recon.sh @@ -0,0 +1,83 @@ +#!/bin/bash +# Tillicum first-login recon. Read-only, ~10 seconds, submits nothing and costs nothing. +# +# Everything in docs/tillicum.md marked UNVERIFIED was written from the public docs +# without an account. This settles those items in one pass so the doc, the dotfiles +# ssh target, and run_yolo_train_tillicum.slurm can be corrected from fact. +# +# ./scripts/tillicum_recon.sh (on a Tillicum login node) +# wsl-ssh.ps1 tillicum script scripts/tillicum_recon.sh (from a Claude session) +# +# CPU-only jobs are prohibited on Tillicum, so this deliberately runs on the LOGIN +# node -- it is all metadata queries, nothing that would warrant an allocation. + +echo "==============================================================" +echo " Tillicum recon -- $(date)" +echo " host: $(hostname) user: $USER" +echo "==============================================================" + +echo +echo "## 1. HOME PATH (the top unknown -- wsl-ssh.ps1 currently guesses)" +echo "home: $(cd ~ && pwd)" +echo "quota:" +( quota -s 2>/dev/null || echo " (quota not available)" ) | sed 's/^/ /' + +echo +echo "## 2. PROJECT STORAGE" +for d in /gpfs/projects/makelab /gpfs/projects /gpfs; do + if [ -d "$d" ]; then + echo "$d exists; writable=$( [ -w "$d" ] && echo yes || echo NO )" + df -h "$d" 2>/dev/null | tail -1 | sed 's/^/ /' + break + else + echo "$d MISSING" + fi +done +echo "contents of /gpfs/projects/makelab:" +ls -la /gpfs/projects/makelab 2>&1 | head -10 | sed 's/^/ /' + +echo +echo "## 3. SCHEDULER (expect: no partitions, QoS-driven)" +echo "-- sinfo --" +sinfo -o "%.14P %.6a %.10l %.6D %.6t %N" 2>&1 | head -10 | sed 's/^/ /' +echo "-- QoS available to us --" +sacctmgr -nP show assoc user="$USER" format=Account,QOS 2>&1 | sed 's/^/ /' +echo "-- do we already have long/wide/urgent? --" +sacctmgr -nP show assoc user="$USER" format=QOS 2>/dev/null | tr ',' '\n' \ + | grep -iE "long|wide|urgent" | sed 's/^/ /' || echo " (none -- long QoS needs the request form)" + +echo +echo "## 4. BILLING" +echo "-- hyakusage --" +( command -v hyakusage >/dev/null && hyakusage 2>&1 || echo " hyakusage NOT on PATH" ) | sed 's/^/ /' + +echo +echo "## 5. SOFTWARE (docs say Apptainer, not modules -- verify)" +for t in apptainer singularity module conda python3 nvidia-smi rsync globus; do + p=$(command -v "$t" 2>/dev/null) + printf " %-12s %s\n" "$t" "${p:-NOT FOUND}" +done +echo "-- apptainer version --" +( apptainer --version 2>&1 || echo " n/a" ) | sed 's/^/ /' +echo "-- modules, if any --" +( module avail 2>&1 | head -15 || echo " n/a" ) | sed 's/^/ /' + +echo +echo "## 6. MAIL (klone forced END,FAIL,TIME_LIMIT via a lua job_submit plugin and" +echo "## ~1 email/min under preemption -- check whether Tillicum does the same)" +scontrol show config 2>/dev/null \ + | grep -iE "JobSubmitPlugins|MailProg|MailDomain|PreemptMode|MaxJobCount" | sed 's/^/ /' + +echo +echo "## 7. NETWORK PATH BACK TO KLONE (286 GB / ~911k files has to cross)" +echo " NOTE: transfer ARCHIVES, not the tree. Per-file copy would pay ~911k round trips." +for h in klone.hyak.uw.edu; do + printf " %-24s " "$h" + ( timeout 5 bash -c "/dev/null && echo "port 22 reachable" ) || echo "port 22 NOT reachable from here" +done + +echo +echo "==============================================================" +echo " Next: update docs/tillicum.md (home path, QoS, mail, software)," +echo " fix Home in wsl-ssh.ps1 \$Targets, then run the debug smoke test." +echo "==============================================================" From fee620e8daa54879c6df9b3937b559ce37c331b9 Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Thu, 30 Jul 2026 10:54:58 -0700 Subject: [PATCH 4/7] Save periodic checkpoints on Tillicum, and document what resume cannot inherit Ultralytics defaults save_period=-1, keeping only last.pt and best.pt. That permanently forecloses any analysis needing a checkpoint from a specific epoch, and it cannot be recovered after the fact -- results.csv has per-epoch val metrics but no weights. The analysis we would lose: RampNet's published model trained ~1 epoch / ~9.4k steps at constant LR (#84) while this baseline gets 60 epochs with a schedule. If RampNet wins anyway that is a stronger result, but only if we can also report a COMPUTE-MATCHED YOLO point rather than just the converged one. Cost is ~150 MB per checkpoint against a 1 TB allocation. Also documents a limitation found while wiring it: resume=True reuses the args saved in the checkpoint, so an arm resumed from a klone checkpoint keeps that run's save_period=-1 and this setting does not apply to it. Per-epoch weights therefore exist only for runs started under this script. Less costly than it sounds -- the compute-matched point we most want is roughly RampNet's one-epoch budget, which is exactly the ep1 best.pt that four of the six klone arms are still holding. Co-Authored-By: Claude Opus 5 (1M context) --- .../run_yolo_train_tillicum.slurm | 31 +++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/scripts/model_comparison/run_yolo_train_tillicum.slurm b/scripts/model_comparison/run_yolo_train_tillicum.slurm index 1f873db..08feaf5 100644 --- a/scripts/model_comparison/run_yolo_train_tillicum.slurm +++ b/scripts/model_comparison/run_yolo_train_tillicum.slurm @@ -64,6 +64,22 @@ YOLO_IMGSZ="${YOLO_IMGSZ:-1024}" EPOCHS="${EPOCHS:-60}" BATCH="${BATCH:--1}" # pin per config to match the klone runs PATIENCE="${PATIENCE:-20}" + +# Keep a checkpoint every N epochs. Ultralytics defaults save_period=-1, keeping ONLY +# last.pt and best.pt -- which forecloses, permanently and retroactively, any analysis +# that needs a checkpoint from a specific epoch. +# +# The one we care about: RampNet's published model was trained for ~1 epoch / ~9.4k +# steps at constant LR (#84), while this baseline gets 60 epochs with a schedule. If +# RampNet wins anyway that is a STRONGER result -- but only if we can also report a +# COMPUTE-MATCHED YOLO point, i.e. the checkpoint at the epoch where YOLO has consumed +# comparable training, not just the converged one. That requires the checkpoint to +# still exist. results.csv gives per-epoch val metrics but no weights, so it cannot be +# reconstructed after the fact. +# +# Cost is disk (~150 MB per checkpoint per arm) against a 1 TB allocation. Cheap +# insurance for an option that cannot be bought back later. +SAVE_PERIOD="${SAVE_PERIOD:-5}" # On klone this defaulted to 0 because ultralytics `time=` restarts its clock on every # preemption. Tillicum never preempts, so `time=` is safe here -- but leave it 0 for # any run meant to be comparable to the #51 klone results, which are epoch-based. @@ -98,14 +114,24 @@ echo "------------------------------------------------" run_train() { "$@" - "$YOLO_CKPT" "$YOLO_DATA" "$YOLO_IMGSZ" "$EPOCHS" "$BATCH" "$PATIENCE" \ - "$PROJECT" "$NAME" "$TRAIN_HOURS" "$DEVICE" <<'PY' + "$PROJECT" "$NAME" "$TRAIN_HOURS" "$DEVICE" "$SAVE_PERIOD" <<'PY' import os, sys from ultralytics import YOLO -ckpt, data, imgsz, epochs, batch, patience, project, name, hours, device = sys.argv[1:11] +(ckpt, data, imgsz, epochs, batch, patience, project, name, hours, device, + save_period) = sys.argv[1:12] last = os.path.join(project, name, "weights", "last.pt") # Still needed on Tillicum, but for a different reason than on klone: not preemption, # but the 24 h normal-QoS ceiling. A 60-epoch tiles schedule does not fit in one job, # so chain jobs and resume. Unlike klone the boundary is predictable, not random. +# +# CAVEAT: resume=True reuses every training arg SAVED IN THE CHECKPOINT, so a run +# resumed from a klone checkpoint keeps that run's save_period=-1 -- SAVE_PERIOD above +# does NOT apply to it. Per-epoch checkpoints therefore exist only for runs STARTED +# under this script. For the klone arms the early-epoch weights are already gone. +# +# That is less costly than it sounds: the compute-matched point we most want (YOLO at +# roughly RampNet's ~1-epoch budget) is exactly the ep1 best.pt that four of the six +# klone arms are still holding. The "one-epoch models" are the matched comparison. if os.path.exists(last): print(f"[resume] {last} exists -> resuming", flush=True) YOLO(last).train(resume=True) @@ -116,6 +142,7 @@ else: kw = dict( data=data, imgsz=int(imgsz), epochs=int(epochs), batch=batch_arg, patience=int(patience), project=project, name=name, device=dev, exist_ok=True, + save_period=int(save_period), ) if float(hours) > 0: kw["time"] = float(hours) From 3d12a7ae0ca19a348b006f6d16d3529e1dc754b6 Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Thu, 30 Jul 2026 17:19:42 -0700 Subject: [PATCH 5/7] Fold in UW-IT's answers: SquashFS, /gpfs/scrubbed, and a safe enforced budget All six questions to UW-IT came back (Sumaiya Sathar, 2026-07-30). Three answers change the plan, so they are recorded in the doc rather than left in a mailbox. Budget is set to $1,500/month with enforcement ACTIVE, and enforcement does not cancel running jobs -- it only blocks new submissions. That removes the only reason we had to prefer warn-only, so the runaway-spend guard is free. There is no way to get CPUs without the matching GPU allocation, and it is a rate-model constraint rather than a scheduling one, so the DEVICE=0 / 2-GPU design in the launcher is now confirmed necessary rather than provisional. The transfer plan changes outright: many-small-file datasets belong on /gpfs/scrubbed rather than the 1 TB project quota, and UW-IT recommends SquashFS. So pack_yolo_dataset.slurm replaces the tar approach -- tar fixed the transfer but not the destination, since untarring recreates all ~911k files on /gpfs and pays the metadata cost permanently. A SquashFS image is mounted read-only and the destination only ever holds one file. The subtle part, called out in both the script and the doc because getting it wrong is invisible: Ultralytics validates labels/.cache against a hash of absolute paths. The .cache files must be inside the image (they exist on klone already, and a read-only mount cannot create them), and mounting at a different path silently invalidates them and forces a full ~911k-file rescan. Training still works -- it is just permanently slow. Also confirmed: 1 TB is correct for us (100 GB applies to demo accounts without a worktag), there is no project end date, storage grows in 1 TB increments on request, and the 100 free GPU hours do not expire. Co-Authored-By: Claude Opus 5 (1M context) --- docs/tillicum.md | 82 +++++++++--- .../model_comparison/pack_yolo_dataset.slurm | 123 ++++++++++++++++++ 2 files changed, 187 insertions(+), 18 deletions(-) create mode 100644 scripts/model_comparison/pack_yolo_dataset.slurm diff --git a/docs/tillicum.md b/docs/tillicum.md index 8865dae..2aa969d 100644 --- a/docs/tillicum.md +++ b/docs/tillicum.md @@ -153,15 +153,27 @@ Two conclusions: `ls -lU | head -201` on `tiles/images/train` timed out. That is metadata throughput, not bandwidth. -**So transfer an archive, never the tree.** `tar` per split on klone → move a handful of -large files → untar on Tillicum. A per-file `rsync`/`scp` would pay ~911,000 round trips. -Run the `tar` inside a Slurm job, **not on a klone login node** — that is exactly the -heavy-login-process reap that kills the SSH master. Note the untar on `/gpfs` is itself -metadata-heavy, so time it: it is our first real measurement of whether Tillicum's flash -actually fixes this, which is the same question the tiles training arm is asking. +**So pack into SquashFS, never copy the tree** — see +`scripts/model_comparison/pack_yolo_dataset.slurm`, and UW-IT's own recommendation in the +answers below. A per-file `rsync`/`scp` pays ~911,000 round trips; `tar` fixes the +transfer but not the destination, since untarring recreates all 911k files on `/gpfs` and +pays the metadata cost permanently. A SquashFS image is mounted read-only, so the +destination only ever holds **one file**. + +Run the pack inside a Slurm job, **not on a klone login node** — that is exactly the +heavy-login-process reap that kills the SSH master. + +**The subtle part is the Ultralytics label cache.** Ultralytics writes +`labels/.cache` and validates it against a hash of the label+image **absolute +paths**. So (a) the `.cache` files must be *inside* the image — they exist on klone +already and a read-only mount cannot create them — and (b) mounting at a different path +than they were built under silently invalidates them and forces a full ~911k-file +rescan. Budget one slow first epoch on Tillicum to regenerate them, and do not mistake +it for the steady-state cost. Getting this wrong is invisible: training still works, it +is just permanently slow. Staging order: **`pano` first (76 GB, 193k files)** — it unblocks the smoke test and the -pano epoch-time measurement while the much larger `tiles` archive moves behind it. +pano epoch-time measurement while the much larger `tiles` image builds behind it. > The docs describe demo accounts as receiving "100 GB of dedicated project storage," > while our provisioning email says 1 TB. Assume 1 TB (the email is specific to us) but @@ -233,17 +245,51 @@ Record the measured epoch times back into this file and into #70. - **Budget changes:** contact UW-IT; enforcement semantics (blocks new submissions vs. cancels running jobs) are **UNVERIFIED** and were asked about on 2026-07-30. -## Open questions - -Sent to UW-IT 2026-07-30, unanswered at time of writing: - -1. Does budget enforcement block *new submissions* or *cancel running jobs*? -2. Is there any way to get more CPUs per job without the GPU billing multiplier? -3. Guidance for many-small-file datasets on `/gpfs`; is packing into an archive or - container format recommended? -4. Storage: quota-increase path, and what "purged at the end of the project" means. -5. Do the 100 free demo hours expire? -6. How to request `long` QoS access. +## Answers from UW-IT (Sumaiya Sathar, 2026-07-30) + +All six questions answered. Several change the plan, so they are recorded here rather +than left in a mailbox. + +1. **Budget enforcement is safe.** Set to **$1,500/month, enforcement active**. Enforced + budgets **do not cancel running jobs** — a job already running continues to + completion; enforcement only **blocks new submissions** until the cap is raised or + the period rolls over. So the cap cannot destroy an in-flight experiment, which was + the only reason to have preferred warn-only. +2. **No CPUs without GPUs — confirmed, and it is a rate-model constraint, not a + scheduling one.** There is no QoS granting extra cores without the matching GPU + allocation; UW-IT is considering it, but the UW-approved rate model cannot charge for + CPUs independently, and unbalanced nodes are undesirable. **So the 2-GPU dataloader + trick is the only lever**, and its 2× billing is unavoidable — see the trap above. +3. **Many small files → `/gpfs/scrubbed`, and pack into SquashFS.** This is the big one. + Active many-small-file datasets belong on **`/gpfs/scrubbed`** (larger capacity, has + an automatic cleanup policy) rather than the 1 TB project quota, and UW-IT explicitly + recommends **SquashFS** for this shape. Data Commons is an option if the dataset can + be public. +4. **Storage grows in 1 TB increments** on request to `help@uw.edu` (subject "Tillicum") + with a workflow justification — it is shared active-compute storage, not archival. +5. **No project end date.** "Purged at the end of the project" simply means that if we + stop using Tillicum we must copy data off so the space can be reclaimed. Nothing + expires on a timer. +6. **1 TB is correct for us** — the 100 GB figure in the public docs applies to *demo* + accounts. Providing a worktag makes it a regular account, which is why we got 1 TB. +7. **The 100 free GPU hours do not expire.** They remain until used. +8. **`long` QoS requires the Special QoS Access Request Form** — still to be submitted, + then reviewed. Not a blocker for the measurement phase (everything fits in 24 h), but + it gates comfortable production runs. + +### What this changes + +- **Dataset goes to `/gpfs/scrubbed`, not `/gpfs/projects/makelab`.** The 1 TB project + quota is for artifacts we want backed up (checkpoints, results), not the 286 GB of + training data. This also removes storage pressure from the plan entirely. +- **Pack with SquashFS, not tar.** See + `scripts/model_comparison/pack_yolo_dataset.slurm`. tar fixes the *transfer* but not + the *destination* — untarring recreates all ~911k files on `/gpfs` and pays the + metadata cost permanently. A SquashFS image is mounted read-only and the destination + only ever holds one file. +- **Enforced budget is safe to leave on**, so the runaway-spend guard costs us nothing. +- **The `DEVICE=0` / 2-GPU design in the Tillicum launcher is now confirmed necessary** + rather than a workaround pending a better answer. ## Related diff --git a/scripts/model_comparison/pack_yolo_dataset.slurm b/scripts/model_comparison/pack_yolo_dataset.slurm new file mode 100644 index 0000000..c8127cc --- /dev/null +++ b/scripts/model_comparison/pack_yolo_dataset.slurm @@ -0,0 +1,123 @@ +#!/bin/bash +# Pack one YOLO dataset into a SquashFS image, for transfer klone -> Tillicum. +# +# WHY SQUASHFS AND NOT rsync/scp/tar +# The dataset is 286 GB in ~911,000 files (tiles 210 GB / 718,415; pano 76 GB / +# 192,938 -- measured 2026-07-30). That is metadata-bound, not bandwidth-bound: `du -sh` +# over the tiles tree beat three timeouts before finishing on a 550 s budget, and even +# `ls -lU | head -201` on tiles/images/train timed out. +# +# - rsync/scp of the tree pays ~911k round trips. No. +# - tar fixes the TRANSFER but not the DESTINATION: untarring recreates all 911k files +# on /gpfs, paying the metadata cost again and permanently. +# - SquashFS pays it ONCE, here, and the destination only ever sees ONE file. The +# training job mounts it read-only and the kernel serves the tree from a compact +# in-image index. This is also what UW-IT recommends for many-small-file datasets +# (Sumaiya, 2026-07-30). +# +# WHY A SLURM JOB AND NOT A LOGIN NODE +# klone reaps heavy login processes, and that reap also kills the SSH control master +# every agent session depends on. Never run this interactively. +# +# WHERE IT WRITES +# /gscratch/scrubbed, NOT /gscratch/makelab: lab storage is 84% full with ~168 GB free +# (checked 2026-07-30) and cannot hold this. scrubbed has ~15 TB free, and its ~21-day +# purge does not matter for a transient staging area. +# +# DS=pano sbatch -A ckpt-makelab scripts/model_comparison/pack_yolo_dataset.slurm +# DS=tiles sbatch -A ckpt-makelab scripts/model_comparison/pack_yolo_dataset.slurm +# +#SBATCH --job-name=yolo_pack +#SBATCH -p ckpt +#SBATCH -q ckpt +#SBATCH --requeue +#SBATCH --time=12:00:00 +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=8 +#SBATCH --mem=32G +#SBATCH --mail-type=NONE +#SBATCH --output=logs/yolo_pack_%j.out +#SBATCH --error=logs/yolo_pack_%j.err + +set -euo pipefail + +DS="${DS:?set DS to pano or tiles}" +SRC="${SRC:-/gscratch/scrubbed/$USER/yolo}" +OUT="${OUT:-/gscratch/scrubbed/$USER/yolo_squashfs}" +mkdir -p "$OUT" + +IMG="$OUT/${DS}.sqfs" +SUM="$IMG.sha256" + +echo "--- pack $DS ---" +echo "src: $SRC/$DS out: $IMG" +df -h "$OUT" | tail -1 + +# Idempotent: a completed, verified image is left alone, so a ckpt requeue or a manual +# resubmit is free rather than a restart from zero. +if [ -f "$IMG" ] && [ -f "$SUM" ]; then + echo "[skip] image exists; verifying..." + if ( cd "$OUT" && sha256sum -c "$(basename "$SUM")" ); then + echo "[skip] verified, nothing to do"; exit 0 + fi + echo "[warn] checksum FAILED -- rebuilding" + rm -f "$IMG" "$SUM" +fi + +# --------------------------------------------------------------------------------- +# THE ULTRALYTICS LABEL CACHE IS THE SUBTLE PART -- READ BEFORE CHANGING THIS. +# +# Ultralytics writes labels/.cache next to the labels and validates it against a +# hash of the label+image FILE PATHS. Two consequences: +# +# 1. The .cache files MUST be inside the image. They already exist on klone (built +# 2026-07-25/26), and a read-only mount cannot create them. Without them every run +# rescans ~911k files -- exactly the cost this whole exercise exists to avoid. +# +# 2. The hash covers absolute paths, so mounting at a DIFFERENT path than the one the +# cache was built under INVALIDATES it and forces that same rescan. Mount the image +# on Tillicum at a stable path and expect to regenerate the cache ONCE there (it +# needs a writable overlay, or a one-off run with labels/ bind-mounted writable). +# Budget one slow first epoch; do not mistake it for the steady-state cost. +# +# This is called out in docs/tillicum.md too, because getting it wrong is invisible -- +# training still works, it is just silently slow forever. +# --------------------------------------------------------------------------------- +echo "[check] label caches that will be packed:" +ls -la "$SRC/$DS/labels/"*.cache 2>/dev/null || echo " NONE -- expect a full rescan on first use" + +# -noD: do NOT compress data blocks. The payload is JPEG, already compressed; deflating +# it would burn hours of CPU for ~nothing and make reads slower. Metadata/inodes are +# still compressed, which is where the win actually is. +# -no-xattrs: nothing here uses them, and they bloat the index. +echo "[mksquashfs] starting $(date)" +mksquashfs "$SRC/$DS" "$IMG" \ + -noD \ + -no-xattrs \ + -processors "${SLURM_CPUS_PER_TASK:-8}" \ + -info +echo "[mksquashfs] done $(date)" + +# The checksum travels with the image and is verified after transfer, same discipline as +# the weight snapshot's MANIFEST.md. +( cd "$OUT" && sha256sum "$(basename "$IMG")" > "$(basename "$SUM")" ) + +ls -lh "$IMG" +cat "$SUM" + +cat < + +Verify the checksum on arrival BEFORE trusting it: + sha256sum -c $(basename "$SUM") +EOF From f8c287e3934cbf955034e1f9df540eb8cae14915 Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Fri, 31 Jul 2026 06:05:09 -0700 Subject: [PATCH 6/7] Regenerate the YOLO dataset on Tillicum, and prove it matches klone The SquashFS plan worked and still did not get the data across. klone job 37940649 packed pano into a single 76 GB image in 2h38m, solving the destination problem exactly as designed. But there is no automated klone -> Tillicum route: no shared filesystem, and neither end can authenticate to the other non-interactively, since both are Duo/keyboard-interactive with no publickey and BatchMode fails in both directions. Globus CLI is on neither side. The transfer, not the destination, was the blocker. So this regenerates the dataset on Tillicum from Hugging Face instead. That is safe rather than merely convenient: prepare_yolo_dataset.py thins background tiles with an md5 of the file stem, specifically so the choice is stable across processes and runs, so the same source and the same flags must reproduce the same split. VERIFIED, and further than counts. Job 198910 (4h40m) reproduced all four counts in the #51 record. Because klone's tree is still live, we could also compare identity rather than cardinality: the md5 of the sorted filename list is identical on both clusters for all eight image and label directories. The two datasets hold the same files under the same train/val split, not merely the same number of them. 767,840 boxes over 192,938 panos, 0 read errors. Anything trained on Tillicum is comparable to the klone arms on the data axis. The launcher is committed as-run, byte-identical to the copy that produced the dataset, per the provenance rule in the yolo_baseline README. That includes a cosmetic quirk on the last line, where backticks inside a double-quoted echo execute hyakusage rather than printing its name. Harmless, and it happens to log the job's cost, but it is an accident and not a design -- left alone because rewriting it would break the byte-identical claim. Two caveats recorded with the numbers. It cost $4.20, because Tillicum rejects CPU-only jobs and a prep job must hold an H200 it never uses -- structural, and the argument for keeping prep on klone whenever the data is reachable there. And it landed on /gpfs/scrubbed, not the 1 TB project quota, so it is subject to the inactivity purge and must not be assumed to survive a gap between arms. Co-Authored-By: Claude Opus 5 (1M context) --- docs/tillicum.md | 59 ++++++++++++ .../run_yolo_data_prep_tillicum.slurm | 92 +++++++++++++++++++ 2 files changed, 151 insertions(+) create mode 100644 scripts/model_comparison/run_yolo_data_prep_tillicum.slurm diff --git a/docs/tillicum.md b/docs/tillicum.md index 2aa969d..e72737c 100644 --- a/docs/tillicum.md +++ b/docs/tillicum.md @@ -179,6 +179,65 @@ pano epoch-time measurement while the much larger `tiles` image builds behind it > while our provisioning email says 1 TB. Assume 1 TB (the email is specific to us) but > confirm. +### How the data actually got there: regenerate, don't transfer (resolved 2026-07-31) + +The SquashFS plan above is sound and it **worked** — klone job `37940649` packed +`pano` into a single 76 GB `pano.sqfs` in 2 h 38 m (sha256 `75be5150…aea1dd0a`, still on +klone at `/gscratch/scrubbed/jfroehli/yolo_squashfs/`). It solved the destination +problem exactly as designed: one file instead of 193k. + +**It did not solve the transfer, and the transfer is the real blocker.** There is no +automated klone → Tillicum path: no shared filesystem (`/gscratch` and `/mmfs1` do not +exist on Tillicum), and neither end can authenticate to the other non-interactively — +both are Duo/keyboard-interactive with no `publickey`, so `BatchMode` fails in both +directions. Globus CLI is installed on neither side. Verified 2026-07-30. + +So we **regenerated the dataset on Tillicum from Hugging Face** instead +(`scripts/model_comparison/run_yolo_data_prep_tillicum.slurm`, job `198910`, 4 h 40 m). +This is safe rather than merely convenient, because the prep is deterministic by +construction: `prepare_yolo_dataset.py` thins background tiles with an md5 of the file +stem specifically so the choice is stable across processes and runs (see +`_keep_background` and its comment about salted `hash()`). + +**Verified equivalent to klone, 2026-07-31.** Counts match the #51 record in all eight +directories — and, because klone's tree is still live, we could go further than counts. +The md5 of the sorted filename list is **identical on both clusters for all eight**, so +the two datasets contain the same files under the same train/val split, not merely the +same number of them: + +| directory | files | md5 of sorted filename list (klone == Tillicum) | +|---|---:|---| +| `tiles/images/train` | 557,413 | `f5e664fbd64651be0ff89045d217ff50` | +| `tiles/images/val` | 161,002 | `f48f91878d34e72a7b0b2dfd48f6c90a` | +| `pano/images/train` | 150,063 | `a8f3af23a61952ef1209435ca0e295ea` | +| `pano/images/val` | 42,875 | `83f5371483f3dbfa5a6aece939b86901` | +| `tiles/labels/train` | 557,413 | `19c5c38ae148a907042d213c84002cc6` | +| `tiles/labels/val` | 161,002 | `711d5a59a2d6019ac1857133b167266e` | +| `pano/labels/train` | 150,063 | `015fc700faee211e37af3f8edeca0dfc` | +| `pano/labels/val` | 42,875 | `2fc039583a9b0cc08ca57b11739f5a5a` | + +Reproduce on either cluster with, per directory: +`ls -U /yolo/ | sort | md5sum` — klone root `/gscratch/scrubbed/jfroehli`, +Tillicum root `/gpfs/scrubbed/jfroehli`. The prep run itself reported 767,840 boxes over +192,938 panos with **0 read errors**. Anything trained on Tillicum is therefore +comparable to the #51 klone arms on the data axis. + +Two caveats that travel with this: + +- **It cost $4.20** — 4.67 GPU-hours at `normal` QoS. Tillicum rejects CPU-only jobs, so + a data-prep job must hold an H200 it never uses. That is structural, not an error, but + it is the argument for keeping prep on klone (free) whenever a dataset already exists + there and *can* be reached. Here it could not be. +- **It landed on `/gpfs/scrubbed`, not the 1 TB project quota** — 1.61 TB across 2.28 M + files, counting the ~462 GB Hugging Face source cache. `scrubbed` is purged on an + inactivity timer, so this copy is not durable: it is fine while an arm is actively + reading it, and must not be assumed to survive a gap between arms. Durable artifacts + still belong on klone's `/gscratch/makelab`. + +Keep the SquashFS path documented anyway: it is the right answer the moment a transfer +route exists (a Globus endpoint, or an intermediate host either end can reach), and the +`pano.sqfs` image is already built. + ## Software environment Tillicum's documented path is **Apptainer containers**, not conda modules — diff --git a/scripts/model_comparison/run_yolo_data_prep_tillicum.slurm b/scripts/model_comparison/run_yolo_data_prep_tillicum.slurm new file mode 100644 index 0000000..92c471b --- /dev/null +++ b/scripts/model_comparison/run_yolo_data_prep_tillicum.slurm @@ -0,0 +1,92 @@ +#!/bin/bash +# Rebuild the YOLO training dataset ON TILLICUM, from Hugging Face. Download + prep. +# +# WHY REGENERATE RATHER THAN TRANSFER FROM KLONE +# There is no automated path from klone to Tillicum: no shared filesystem (/gscratch and +# /mmfs1 do not exist here), and Tillicum cannot authenticate to klone non-interactively +# — both ends are Duo/keyboard-interactive with no publickey, so BatchMode fails. Globus +# CLI is on neither side. Verified 2026-07-30. +# +# Regenerating sidesteps all of it, and is SAFE because the prep is deterministic by +# construction: prepare_yolo_dataset.py thins background tiles with an md5 of the file +# stem specifically so the choice is stable across processes and runs (see +# `_keep_background`, and its comment about salted hash()). +# +# VERIFY, DO NOT ASSUME. The committed record gives the exact counts klone produced: +# tiles 557,413 train / 161,002 val pano 150,063 train / 42,875 val +# This job prints its own counts at the end. If they do not match, the datasets are NOT +# equivalent and nothing trained here is comparable to the #51 runs — stop and diff. +# +# WHY A GPU JOB FOR CPU WORK +# Tillicum prohibits CPU-only jobs; every job must request >=1 GPU. The runbook also +# warns that login nodes reap heavy long-running processes, and on Tillicum that would +# take the Duo-authenticated SSH master with it. So we hold one GPU and eat the cost: +# roughly 6-12 h at $0.90/h, i.e. $5-11. Cheap against a $1,500/month cap. +# +# sbatch scripts/model_comparison/run_yolo_data_prep_tillicum.slurm +# +#SBATCH --job-name=yolo_data_prep +#SBATCH --qos=normal +#SBATCH --gres=gpu:1 +#SBATCH --cpus-per-task=8 +#SBATCH --mem=200G +#SBATCH --time=12:00:00 +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --mail-type=NONE +#SBATCH --output=logs/yolo_data_prep_%j.out +#SBATCH --error=logs/yolo_data_prep_%j.err + +set -euo pipefail + +export REPO="${REPO:-/gpfs/home/$USER/RampNet}" +# /gpfs/scrubbed, NOT the 1 TB /gpfs/projects/makelab quota: the HF source is ~462 GB +# and the prepared YOLO layout another ~286 GB, so this needs ~750 GB. scrubbed has +# ~484 TB free and is what UW-IT recommends for active many-small-file datasets. +export SCRATCH="${SCRATCH:-/gpfs/scrubbed/$USER}" +export PYBIN="${PYBIN:-/gpfs/projects/makelab/$USER/envs/rampnet-yolo/bin/python}" +export HF_HOME="$SCRATCH/hf" +export YOLO_CONFIG_DIR="$SCRATCH/ultralytics" + +mkdir -p "$SCRATCH" + +echo "=== node ===" +hostname; date +df -h "$SCRATCH" | tail -1 +echo "=== interpreter (must match the klone #51 runs exactly) ===" +"$PYBIN" -c "import sys, torch, ultralytics; print('python', sys.version.split()[0], '| torch', torch.__version__, '| ultralytics', ultralytics.__version__)" + +cd "$REPO" + +echo +echo "=== STAGE: data (HF download, ~462 GB) ===" +date +bash hyak_yolo_runbook.sh data + +echo +echo "=== STAGE: prep (tiles + pano) ===" +date +bash hyak_yolo_runbook.sh prep + +echo +echo "=== VERIFICATION — counts must match the klone record ===" +date +for ds in tiles pano; do + for split in train val; do + d="$SCRATCH/yolo/$ds/images/$split" + printf " %-6s %-6s %s\n" "$ds" "$split" "$(ls -U "$d" 2>/dev/null | grep -c '\.jpg$')" + done +done +cat <<'EOF' + + Expected, from scripts/model_comparison/yolo_baseline/README.md provenance: + tiles train 557413 + tiles val 161002 + pano train 150063 + pano val 42875 + + A mismatch means the regenerated dataset is NOT the one the #51 arms trained on. + Do not train against it until the difference is understood. +EOF +date +echo "Cost so far: run `hyakusage`" From fa469464abcd5e22a8db305ed82c18ce0b0c877b Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Fri, 31 Jul 2026 06:39:50 -0700 Subject: [PATCH 7/7] Commit the Tillicum env-setup and smoke scripts, with what they measured Both ran; neither was checked in. tillicum_smoke.slurm is committed byte-identical to the copy that ran as job 198638 (md5 687cdc31, 5042 bytes). WHAT THE ENV SCRIPT PROVED. It reproduces the klone #51 toolchain exactly -- verified by running the interpreter, not by reading a lockfile: 3.11.15 / torch 2.13.0+cu126 / ultralytics 8.4.105, character-for-character what klone's training logs report. It has since driven a full 4h40m production job without incident, so the toolchain axis of comparability is closed the same way the data axis now is. The non-obvious part, recorded in the doc: conda is not a preference here. Tillicum's system python is 3.9.25 and the cu126 index tops out at torch 2.8.0 for 3.9, so a naive pip install silently yields 2.8.0 + 8.4.113 -- a baseline nobody can publish, arrived at with no error message. The script's version check is a gate rather than a report: it refuses to install a substitute and exits non-zero, which is what catches ultralytics pulling its own torch over the pinned one during its dependency resolution. Note this does NOT verify environment.yml, which is a different stack (CUDA 11.8, conda-forge, linux-64) that nothing on the YOLO path needs. That entry stays UNVERIFIED rather than being quietly upgraded. A DISCREPANCY THE SMOKE SCRIPT'S HEADER CANNOT SEE. It asserts the debug QoS "COSTS NOTHING," and sacctmgr agrees: UsageFactor is exactly 0.000000, with priority 50 against normal's 25, so probes are both free and faster to start. But hyakusage disagrees -- it reports the 2-minute smoke job as 0.03 GPU-hours and $0.03, which is raw wall-clock x $0.90 with the zero multiplier apparently not applied, contradicting its own header line about billable hours. We do not know which one ITBill follows. Left as a documented open question rather than a silent edit, since the script is the as-run record and its sacctmgr citation is accurate as far as it goes. The exposure is bounded at $0.90 per job either way, because debug caps at 1 h and 1 GPU, so this is worth asking UW-IT but not worth blocking on. The full QoS table is now in the doc so the claim can be rechecked without cluster access. Co-Authored-By: Claude Opus 5 (1M context) --- docs/tillicum.md | 61 ++++++++++++++++- scripts/tillicum_setup_env.sh | 91 ++++++++++++++++++++++++++ scripts/tillicum_smoke.slurm | 120 ++++++++++++++++++++++++++++++++++ 3 files changed, 270 insertions(+), 2 deletions(-) create mode 100644 scripts/tillicum_setup_env.sh create mode 100644 scripts/tillicum_smoke.slurm diff --git a/docs/tillicum.md b/docs/tillicum.md index e72737c..1675f41 100644 --- a/docs/tillicum.md +++ b/docs/tillicum.md @@ -109,6 +109,33 @@ At that ceiling: Monitor with `hyakusage`. +### `debug` is free — but the two cost tools disagree (measured 2026-07-31) + +The full QoS table, from `sacctmgr show qos`: + +| QoS | Priority | UsageFactor | MaxWall | per-user cap | +|---|---:|---:|---|---| +| `debug` | 50 | **0.000000** | 1 h | 1 GPU, 8 CPU, 200 G, 1 node | +| `normal` | 25 | 1.000000 | 1 day | 48 GPU | +| `interactive` | 35 | 1.000000 | 8 h | — | +| `urgent` | 200 | 1.000000 | 3 days | 64 GPU | +| `long` | 25 | 1.000000 | 7 days | — | +| `wide` | 25 | 1.000000 | 1 day | 96 GPU | + +`debug` bills at **UsageFactor 0** and carries *higher* priority than `normal` (50 vs +25), so every environment and throughput probe belongs there. That is the basis for the +"costs nothing" claim in `scripts/tillicum_smoke.slurm`. + +**Caveat, unresolved:** `hyakusage` does not agree. The smoke job (`198638`, 2 min on +`debug`) shows up in its QoS breakdown as **0.03 GPU-hours, $0.03** — i.e. raw +wall-clock × $0.90 with the 0.0 UsageFactor apparently *not* applied, even though +`hyakusage`'s own header says "billable GPU hours = raw GPU hours × QOS multiplier." +Slurm's accounting config and the reporting tool are stating different things, and we do +not know which one ITBill actually follows. Worth asking UW-IT, but not urgent: `debug` +is capped at 1 h × 1 GPU, so the exposure is **at most $0.90 per job** even if +`hyakusage` turns out to be the honest one. Do not quote a "free" figure from this +without saying which tool it came from. + ### The 2-GPU trap for I/O-bound arms This is the cost decision specific to our workload. Our tiles arm is **I/O-bound, not @@ -249,8 +276,38 @@ are DockerHub and the NVIDIA NGC catalog. **Our `environment.yml` should not be assumed to transfer.** It pins linux-64 packages against **CUDA 11.8**, and Tillicum is H200 (sm_90) on Rocky 9. CUDA 11.8 nominally covers sm_90, but a stack built for klone's OS and driver is not a safe bet on a -different distro and a newer card. **UNVERIFIED — do not plan around either outcome -until someone tries it.** The low-risk path is an NGC PyTorch container. +different distro and a newer card. **Still UNVERIFIED** — nobody has tried to solve +`environment.yml` here, because the YOLO baseline does not need it. The low-risk path +remains an NGC PyTorch container. + +### The YOLO stack, however, is VERIFIED (2026-07-31) + +`scripts/tillicum_setup_env.sh` builds it, and it reproduces the klone `#51` toolchain +**exactly** — confirmed by running the interpreter, not by reading a lockfile: + +``` +3.11.15 torch 2.13.0+cu126 ultralytics 8.4.105 +``` + +which is character-for-character what klone's training logs report +(`Ultralytics 8.4.105 · Python-3.11.15 · torch-2.13.0+cu126`). It lives at +`/gpfs/projects/makelab/$USER/envs/rampnet-yolo` — the backed-up 1 TB allocation rather +than `scrubbed`, because the environment is small and annoying to rebuild while the +dataset is huge and reproducible. It has since driven a full 4 h 40 m production job +(`198910`) without incident. + +Two details in that script worth keeping if it is ever edited: + +- **Conda is not optional here.** Tillicum's *system* python is 3.9.25, and the cu126 + wheel index tops out at torch 2.8.0 for 3.9 — so a naive `pip install torch + ultralytics` silently yields 2.8.0 + 8.4.113 and a baseline nobody can publish. The + conda module is the only way to get 3.11 on this cluster. +- **The version check is a gate, not a report.** The script refuses to install a + substitute torch and exits non-zero on any mismatch. That matters because installing + `ultralytics` last can pull its own torch over the pinned one; the post-install assert + is what catches it. A mismatched toolchain produces plausible numbers and an + unpublishable comparison, which is the failure mode `#71`'s protocol exists to + prevent. ## Migrating our Slurm scripts diff --git a/scripts/tillicum_setup_env.sh b/scripts/tillicum_setup_env.sh new file mode 100644 index 0000000..f1dbd2c --- /dev/null +++ b/scripts/tillicum_setup_env.sh @@ -0,0 +1,91 @@ +#!/bin/bash +# Build a Tillicum training environment that MATCHES the klone #51 runs exactly. +# +# WHY EXACT PINS AND NOT "latest" +# The whole point of the Tillicum move is to finish the supervised-YOLO baseline so it +# can be compared against RampNet without a "you undertrained the baseline" objection. +# Resuming klone checkpoints into a different torch/ultralytics, or comparing a +# Tillicum-trained arm against a klone-trained one across library versions, reintroduces +# exactly the uncontrolled variable that #71's protocol exists to eliminate. +# +# The reference is not the README -- it is the training job logs themselves +# (logs/yolo_train_37745363.out on klone), which report: +# +# Python-3.11.15 Ultralytics 8.4.105 torch-2.13.0+cu126 +# +# A naive `pip install torch ultralytics` on Tillicum gives torch 2.8.0 + ultralytics +# 8.4.113, because Tillicum's SYSTEM python is 3.9.25 and the cu126 index tops out at +# 2.8.0 for 3.9. Hence the conda module: it is the only way to get 3.11 here. +# +# WHERE IT LIVES +# /gpfs/projects/makelab -- the 1 TB backed-up allocation, not /gpfs/scrubbed. The +# dataset belongs on scrubbed (it is huge and reproducible); the environment is small +# and annoying to rebuild, so it belongs where there are backups. +# +# wsl-ssh.ps1 tillicum script scripts/tillicum_setup_env.sh + +set -euo pipefail + +PYVER="3.11.15" +TORCH="2.13.0+cu126" +ULTRA="8.4.105" +TORCH_INDEX="https://download.pytorch.org/whl/cu126" + +ENVROOT="${ENVROOT:-/gpfs/projects/makelab/$USER/envs}" +ENVDIR="$ENVROOT/rampnet-yolo" + +echo "=== target: python $PYVER / torch $TORCH / ultralytics $ULTRA ===" +mkdir -p "$ENVROOT" + +module load conda +conda --version + +if [ ! -x "$ENVDIR/bin/python" ]; then + echo "=== creating conda env at $ENVDIR ===" + conda create -y -p "$ENVDIR" "python=$PYVER" +else + echo "=== env exists, reusing $ENVDIR ===" +fi + +PY="$ENVDIR/bin/python" +"$PY" -V + +echo "=== is the pinned torch actually available for this interpreter? ===" +# Fail loudly HERE rather than silently installing a different version: a mismatched +# torch is the kind of thing that produces plausible numbers and an unpublishable +# comparison. +if ! "$PY" -m pip index versions torch --index-url "$TORCH_INDEX" 2>&1 | grep -q "${TORCH}"; then + echo "!! torch $TORCH NOT available for $("$PY" -V). Available:" + "$PY" -m pip index versions torch --index-url "$TORCH_INDEX" 2>&1 | head -3 + echo "!! Refusing to install a substitute -- that would break comparability with #51." + exit 1 +fi + +echo "=== installing pinned torch ===" +"$PY" -m pip install --upgrade pip +"$PY" -m pip install "torch==$TORCH" --index-url "$TORCH_INDEX" + +echo "=== installing pinned ultralytics ===" +# --no-deps would risk a broken install; instead pin ultralytics and let it resolve its +# own deps, then assert torch was not silently upgraded underneath us. +"$PY" -m pip install "ultralytics==$ULTRA" + +echo "=== VERIFY (this is the gate, not the install) ===" +"$PY" - </dev/null | tail -1 | awk '{printf "size=%s avail=%s\n", $2, $4}' + else + printf "%-32s MISSING\n" "$d" + fi +done + +say "SQUASHFS SUPPORT — can we mount the packed dataset at all?" +for t in mksquashfs unsquashfs squashfuse apptainer; do + printf " %-12s %s\n" "$t" "$(command -v $t 2>/dev/null || echo NOT_FOUND)" +done +echo " kernel squashfs: $(grep -qw squashfs /proc/filesystems && echo yes || echo 'no (use apptainer --bind image-src=/)')" + +say "CONDA MODULE PATH" +module load conda 2>&1 | head -5 +command -v conda && conda --version + +# Build somewhere durable and shared, not node-local /tmp. Keep it out of the 286 GB +# dataset's way: envs are small but numerous-filed, so scrubbed is the better home. +ENVROOT="${ENVROOT:-/gpfs/scrubbed/$USER/envs}" +if ! mkdir -p "$ENVROOT" 2>/dev/null; then + echo " (could not write $ENVROOT -- falling back to the project dir)" + ENVROOT="/gpfs/projects/makelab/$USER/envs" + mkdir -p "$ENVROOT" +fi +VENV="$ENVROOT/ultra" +echo "env root: $ENVROOT" + +say "BUILD A MINIMAL ULTRALYTICS ENV" +# A venv on top of the module python is far lighter than solving environment.yml, and +# the only question that matters here is whether a CUDA-12 torch wheel drives an H200. +if [ ! -x "$VENV/bin/python" ]; then + python3 -m venv "$VENV" 2>&1 | tail -2 +fi +"$VENV/bin/python" -m pip install -q --upgrade pip 2>&1 | tail -2 +# cu126 wheels: Tillicum is sm_90, and our klone stack is torch 2.13.0+cu126 (see the +# yolo_baseline record), so matching it keeps the comparison honest. +"$VENV/bin/python" -m pip install -q torch --index-url https://download.pytorch.org/whl/cu126 2>&1 | tail -3 +"$VENV/bin/python" -m pip install -q ultralytics 2>&1 | tail -3 + +say "DOES IT SEE THE GPU?" +"$VENV/bin/python" - <<'PY' +import sys +try: + import torch + print("torch:", torch.__version__, "| cuda build:", torch.version.cuda) + print("cuda available:", torch.cuda.is_available()) + if torch.cuda.is_available(): + print("device:", torch.cuda.get_device_name(0)) + print("capability:", torch.cuda.get_device_capability(0), "(expect (9,0) for H200)") + import time + a = torch.randn(8192, 8192, device="cuda", dtype=torch.bfloat16) + torch.cuda.synchronize(); t0 = time.time() + for _ in range(50): + a @ a + torch.cuda.synchronize() + dt = time.time() - t0 + tflops = 50 * 2 * 8192**3 / dt / 1e12 + print(f"bf16 matmul: {tflops:.0f} TFLOP/s ({dt:.2f}s for 50 iters)") + print(" ref: L40S ~180 dense. A large ratio here is the H200 speedup we are betting on.") +except Exception as e: + print("TORCH FAILED:", type(e).__name__, e); sys.exit(0) + +try: + from ultralytics import YOLO + import ultralytics + print("ultralytics:", ultralytics.__version__) + m = YOLO("yolo11n.yaml") # from config, no download needed + print("model constructed OK") +except Exception as e: + print("ULTRALYTICS FAILED:", type(e).__name__, e) +PY + +say "SLURM VIEW OF THIS JOB (confirm debug really is free)" +scontrol show job "$SLURM_JOB_ID" | grep -oE "QOS=[a-z]+|Partition=[a-z0-9-]+|TRES=[^ ]+" | head -5 + +say "DONE" +date +echo "Check cost with: hyakusage (this job should add 0.00)"