diff --git a/docs/tillicum.md b/docs/tillicum.md new file mode 100644 index 0000000..1675f41 --- /dev/null +++ b/docs/tillicum.md @@ -0,0 +1,415 @@ +# Tillicum — access, migration, and what it costs + +Working notes for running RampNet jobs on **Tillicum**, UW-IT's usage-billed GPU +cluster. The makelab group was provisioned on **2026-07-29**. + +This doc exists because Tillicum is **not** klone-with-different-hostnames — the +scheduler model, the walltime ceiling, the CPU:GPU ratio, and the cost model all +differ in ways that make our existing `.slurm` scripts **non-submittable as written**. +See [Migrating our Slurm scripts](#migrating-our-slurm-scripts) for the diff. + +> **Sourcing.** Everything in the tables below is from the [Tillicum +> docs](https://hyak.uw.edu/docs/systems/tillicum/scheduling-jobs) and the provisioning +> email (2026-07-29), read on 2026-07-30. Sections marked **UNVERIFIED** are our +> inference or are undocumented — we have not run a single job on Tillicum yet, so +> nothing here is confirmed by experience. Correct this file from measurement, not +> from re-reading the docs. + +## Why we care: the klone problem this solves + +The supervised-YOLO baseline (#51) ran on klone's `ckpt` scavenger partition, which is +free but **preemptable**. As of 2026-07-30 that had stopped working as a compute +strategy: + +- **496.5 GPU-hours consumed** on the baseline since 2026-07-24 (`sacct`, all arms). +- **170 job restarts in one night** across five jobs (39 / 37 / 33 / 33 / 28). +- **Zero completed epochs** on five of six arms over that same night; four of the six + arms are still "one-epoch models" holding an ep1 `best.pt`. + +The binding constraint is not the nominal 8.24 h checkpoint slice — it is the +**effective** contiguous run the partition hands out, which collapsed to minutes. Any +arm whose epoch exceeds that interval can never complete one. See #51 and #70. + +**Tillicum does not preempt.** The docs state that even the `urgent` QoS will "not +cancel or preempt jobs that are already running." That is the entire reason to pay. + +## Access + +```bash +ssh @tillicum.hyak.uw.edu +``` + +Duo 2FA, same as klone — **Claude cannot authenticate this.** The `wsl-ssh.ps1` helper +in `dotfiles` has no `tillicum` target yet; adding one (with a `ControlPersist` master, +mirroring the `klone` block in `ssh_config.wsl`) is a prerequisite for driving Tillicum +from an agent session. Until then, Tillicum is Jon-only. + +> The get-started page writes the hostname two ways — `tillicum.hyak.uw.edu` in the +> hostname field and `tillicum.hyak.edu` in the `ssh` example. The former is almost +> certainly correct; **UNVERIFIED** until someone logs in. + +## The scheduler model: QoS, not partitions + +klone is a condo: you get partitions tied to hardware your group owns or scavenges +(`-p ckpt-g2 -q ckpt-gpu`). **Tillicum has no partitions at all.** You pick a QoS, and +billing handles the rest. + +| QoS | Max time | Max GPUs/job | Concurrent | Notes | +|---|---|---|---|---| +| `normal` | **24 h** | 16 | 48 GPUs | default | +| `debug` | 1 h | 1 | 1 job | smoke tests | +| `interactive` | 8 h | 2 | 2 jobs | `salloc` work | +| `long` | **7 days** | 16 | 96 GPUs (shared) | **pre-approval required** | +| `wide` | 24 h | unlimited | 96 GPUs (shared) | **pre-approval required** | +| `urgent` | 3 days | 64 | 96 GPUs (shared) | **pre-approval required** | + +**The 24 h ceiling on `normal` is a real constraint for us.** Our tiles epochs ran +4.1–9.5 h on klone; a 60-epoch schedule does not fit in 24 h. Two options: + +1. **Request `long` QoS** (7 days) via the Tillicum Special QoS Access Request Form. + This is the clean answer and should be requested now — it gates the #70 rerun. +2. **Chain 24 h `normal` jobs**, resuming from `weights/last.pt`. Our + `run_yolo_train.slurm` already has the resume block for this, and without + preemption a resume boundary is predictable rather than random. + +Option 2 works today and needs no approval; option 1 is less operationally fiddly. +Do 1, fall back to 2. + +## Hardware + +192 × NVIDIA **H200** (141 GB HBM3e, NVLink 4.0), 8 GPUs per node, 24 × Dell XE9680, +1,536 Intel Emerald Rapids cores, 400 Gbps NDR InfiniBand, ~3 PB flash, Rocky 9. + +Per-GPU binding: **200 GB system RAM and 8 CPUs**. This ratio is not negotiable — +the docs are explicit that exceeding 8 CPUs per GPU requires requesting *additional +GPUs*. **CPU-only jobs are prohibited**; every job must request ≥1 GPU. + +> Operational consequence: our CPU-side data prep (`run_yolo_data.slurm`, +> `run_yolo_prep.slurm`) has no business on Tillicum. Keep prep on klone, train on +> Tillicum. + +## Cost model + +**$0.90 per GPU-hour**, where **GPU-hour = elapsed wall-clock × N GPUs**. Billed +monthly via ITBill to the lab's UW worktag. (**The worktag and subscription IDs are +deliberately not recorded in this repo** — it is public. Ask Jon, or see the Tillicum +provisioning email.) + +- **100 free GPU demo hours** on the new account (≈ $90). Whether these expire is + **UNVERIFIED**. +- Budget requested 2026-07-30: **$1,500/month** ≈ **1,666 GPU-hours/month**. + +At that ceiling: + +| job shape | wall-clock hours/month | +|---|---| +| 1 GPU | ~1,666 | +| 2 GPUs | ~833 | +| 8 GPUs (full node) | ~208 | + +Monitor with `hyakusage`. + +### `debug` is free — but the two cost tools disagree (measured 2026-07-31) + +The full QoS table, from `sacctmgr show qos`: + +| QoS | Priority | UsageFactor | MaxWall | per-user cap | +|---|---:|---:|---|---| +| `debug` | 50 | **0.000000** | 1 h | 1 GPU, 8 CPU, 200 G, 1 node | +| `normal` | 25 | 1.000000 | 1 day | 48 GPU | +| `interactive` | 35 | 1.000000 | 8 h | — | +| `urgent` | 200 | 1.000000 | 3 days | 64 GPU | +| `long` | 25 | 1.000000 | 7 days | — | +| `wide` | 25 | 1.000000 | 1 day | 96 GPU | + +`debug` bills at **UsageFactor 0** and carries *higher* priority than `normal` (50 vs +25), so every environment and throughput probe belongs there. That is the basis for the +"costs nothing" claim in `scripts/tillicum_smoke.slurm`. + +**Caveat, unresolved:** `hyakusage` does not agree. The smoke job (`198638`, 2 min on +`debug`) shows up in its QoS breakdown as **0.03 GPU-hours, $0.03** — i.e. raw +wall-clock × $0.90 with the 0.0 UsageFactor apparently *not* applied, even though +`hyakusage`'s own header says "billable GPU hours = raw GPU hours × QOS multiplier." +Slurm's accounting config and the reporting tool are stating different things, and we do +not know which one ITBill actually follows. Worth asking UW-IT, but not urgent: `debug` +is capped at 1 h × 1 GPU, so the exposure is **at most $0.90 per job** even if +`hyakusage` turns out to be the honest one. Do not quote a "free" figure from this +without saying which tool it came from. + +### The 2-GPU trap for I/O-bound arms + +This is the cost decision specific to our workload. Our tiles arm is **I/O-bound, not +GPU-bound** — the same epoch took 4.1–9.5 h on klone depending only on which node it +landed on. The obvious fix is more dataloader workers, but on Tillicum **more CPUs +means more GPUs**, and billing multiplies by GPU count: + +- 1 GPU → 8 dataloader CPUs → 1× billing rate +- 2 GPUs → 16 dataloader CPUs → **2× billing rate** + +So 2 GPUs is only cheaper *per epoch* if 16 CPUs makes the tiles epoch **more than +twice** as fast. That is an empirical question, and it is the single best use of the +free demo hours. See [First runs](#first-runs-what-to-measure). + +## Storage + +`/gpfs/projects/makelab` — **1 TB**, backed up daily, "purged at the end of the +project." Note this is *not* archival: `/gscratch/makelab` on klone remains the system +of record for durable weights (see `scripts/model_comparison/yolo_baseline/README.md`). +Keep the existing pattern: **train on Tillicum, `rsync` `best.pt` back to +`/gscratch/makelab`.** + +Undocumented and pending a reply from UW-IT: the quota-increase path, what defines +"end of project," and guidance for **many-small-file** access patterns. That last one +matters to us more than bandwidth. + +### What actually has to move (measured 2026-07-30 on klone) + +| dataset | size | train files | val files | +|---|---|---|---| +| `yolo/tiles` | **210 GB** | 557,413 | 161,002 | +| `yolo/pano` | **76 GB** | 150,063 | 42,875 | +| **total** | **286 GB** | **~911,000 files**, ~300 KB average | | + +Two conclusions: + +- **286 GB fits the 1 TB allocation** with room for checkpoints and run dirs. Storage + quota is not a blocker for the #70 rerun. +- **~911,000 files is the blocker.** This is the many-small-file pathology in its pure + form, and it is not hypothetical: `du -sh` over the tiles tree exceeded a 2-minute + timeout three separate times before completing on a 550 s budget, and even + `ls -lU | head -201` on `tiles/images/train` timed out. That is metadata throughput, + not bandwidth. + +**So pack into SquashFS, never copy the tree** — see +`scripts/model_comparison/pack_yolo_dataset.slurm`, and UW-IT's own recommendation in the +answers below. A per-file `rsync`/`scp` pays ~911,000 round trips; `tar` fixes the +transfer but not the destination, since untarring recreates all 911k files on `/gpfs` and +pays the metadata cost permanently. A SquashFS image is mounted read-only, so the +destination only ever holds **one file**. + +Run the pack inside a Slurm job, **not on a klone login node** — that is exactly the +heavy-login-process reap that kills the SSH master. + +**The subtle part is the Ultralytics label cache.** Ultralytics writes +`labels/.cache` and validates it against a hash of the label+image **absolute +paths**. So (a) the `.cache` files must be *inside* the image — they exist on klone +already and a read-only mount cannot create them — and (b) mounting at a different path +than they were built under silently invalidates them and forces a full ~911k-file +rescan. Budget one slow first epoch on Tillicum to regenerate them, and do not mistake +it for the steady-state cost. Getting this wrong is invisible: training still works, it +is just permanently slow. + +Staging order: **`pano` first (76 GB, 193k files)** — it unblocks the smoke test and the +pano epoch-time measurement while the much larger `tiles` image builds behind it. + +> The docs describe demo accounts as receiving "100 GB of dedicated project storage," +> while our provisioning email says 1 TB. Assume 1 TB (the email is specific to us) but +> confirm. + +### How the data actually got there: regenerate, don't transfer (resolved 2026-07-31) + +The SquashFS plan above is sound and it **worked** — klone job `37940649` packed +`pano` into a single 76 GB `pano.sqfs` in 2 h 38 m (sha256 `75be5150…aea1dd0a`, still on +klone at `/gscratch/scrubbed/jfroehli/yolo_squashfs/`). It solved the destination +problem exactly as designed: one file instead of 193k. + +**It did not solve the transfer, and the transfer is the real blocker.** There is no +automated klone → Tillicum path: no shared filesystem (`/gscratch` and `/mmfs1` do not +exist on Tillicum), and neither end can authenticate to the other non-interactively — +both are Duo/keyboard-interactive with no `publickey`, so `BatchMode` fails in both +directions. Globus CLI is installed on neither side. Verified 2026-07-30. + +So we **regenerated the dataset on Tillicum from Hugging Face** instead +(`scripts/model_comparison/run_yolo_data_prep_tillicum.slurm`, job `198910`, 4 h 40 m). +This is safe rather than merely convenient, because the prep is deterministic by +construction: `prepare_yolo_dataset.py` thins background tiles with an md5 of the file +stem specifically so the choice is stable across processes and runs (see +`_keep_background` and its comment about salted `hash()`). + +**Verified equivalent to klone, 2026-07-31.** Counts match the #51 record in all eight +directories — and, because klone's tree is still live, we could go further than counts. +The md5 of the sorted filename list is **identical on both clusters for all eight**, so +the two datasets contain the same files under the same train/val split, not merely the +same number of them: + +| directory | files | md5 of sorted filename list (klone == Tillicum) | +|---|---:|---| +| `tiles/images/train` | 557,413 | `f5e664fbd64651be0ff89045d217ff50` | +| `tiles/images/val` | 161,002 | `f48f91878d34e72a7b0b2dfd48f6c90a` | +| `pano/images/train` | 150,063 | `a8f3af23a61952ef1209435ca0e295ea` | +| `pano/images/val` | 42,875 | `83f5371483f3dbfa5a6aece939b86901` | +| `tiles/labels/train` | 557,413 | `19c5c38ae148a907042d213c84002cc6` | +| `tiles/labels/val` | 161,002 | `711d5a59a2d6019ac1857133b167266e` | +| `pano/labels/train` | 150,063 | `015fc700faee211e37af3f8edeca0dfc` | +| `pano/labels/val` | 42,875 | `2fc039583a9b0cc08ca57b11739f5a5a` | + +Reproduce on either cluster with, per directory: +`ls -U /yolo/ | sort | md5sum` — klone root `/gscratch/scrubbed/jfroehli`, +Tillicum root `/gpfs/scrubbed/jfroehli`. The prep run itself reported 767,840 boxes over +192,938 panos with **0 read errors**. Anything trained on Tillicum is therefore +comparable to the #51 klone arms on the data axis. + +Two caveats that travel with this: + +- **It cost $4.20** — 4.67 GPU-hours at `normal` QoS. Tillicum rejects CPU-only jobs, so + a data-prep job must hold an H200 it never uses. That is structural, not an error, but + it is the argument for keeping prep on klone (free) whenever a dataset already exists + there and *can* be reached. Here it could not be. +- **It landed on `/gpfs/scrubbed`, not the 1 TB project quota** — 1.61 TB across 2.28 M + files, counting the ~462 GB Hugging Face source cache. `scrubbed` is purged on an + inactivity timer, so this copy is not durable: it is fine while an arm is actively + reading it, and must not be assumed to survive a gap between arms. Durable artifacts + still belong on klone's `/gscratch/makelab`. + +Keep the SquashFS path documented anyway: it is the right answer the moment a transfer +route exists (a Globus endpoint, or an intermediate host either end can reach), and the +`pano.sqfs` image is already built. + +## Software environment + +Tillicum's documented path is **Apptainer containers**, not conda modules — +see [UWrc/tillicum-containers](https://github.com/UWrc/tillicum-containers), which +ships a `pytorch_timm.def` example (we use `timm` for the ConvNeXt-V2 backbone) plus +`run_inference.slurm` and `array_inference.slurm` templates. Recommended image sources +are DockerHub and the NVIDIA NGC catalog. + +**Our `environment.yml` should not be assumed to transfer.** It pins linux-64 packages +against **CUDA 11.8**, and Tillicum is H200 (sm_90) on Rocky 9. CUDA 11.8 nominally +covers sm_90, but a stack built for klone's OS and driver is not a safe bet on a +different distro and a newer card. **Still UNVERIFIED** — nobody has tried to solve +`environment.yml` here, because the YOLO baseline does not need it. The low-risk path +remains an NGC PyTorch container. + +### The YOLO stack, however, is VERIFIED (2026-07-31) + +`scripts/tillicum_setup_env.sh` builds it, and it reproduces the klone `#51` toolchain +**exactly** — confirmed by running the interpreter, not by reading a lockfile: + +``` +3.11.15 torch 2.13.0+cu126 ultralytics 8.4.105 +``` + +which is character-for-character what klone's training logs report +(`Ultralytics 8.4.105 · Python-3.11.15 · torch-2.13.0+cu126`). It lives at +`/gpfs/projects/makelab/$USER/envs/rampnet-yolo` — the backed-up 1 TB allocation rather +than `scrubbed`, because the environment is small and annoying to rebuild while the +dataset is huge and reproducible. It has since driven a full 4 h 40 m production job +(`198910`) without incident. + +Two details in that script worth keeping if it is ever edited: + +- **Conda is not optional here.** Tillicum's *system* python is 3.9.25, and the cu126 + wheel index tops out at torch 2.8.0 for 3.9 — so a naive `pip install torch + ultralytics` silently yields 2.8.0 + 8.4.113 and a baseline nobody can publish. The + conda module is the only way to get 3.11 on this cluster. +- **The version check is a gate, not a report.** The script refuses to install a + substitute torch and exits non-zero on any mismatch. That matters because installing + `ultralytics` last can pull its own torch over the pinned one; the post-install assert + is what catches it. A mismatched toolchain produces plausible numbers and an + unpublishable comparison, which is the failure mode `#71`'s protocol exists to + prevent. + +## Migrating our Slurm scripts + +Our current header (`scripts/model_comparison/run_yolo_train.slurm`) against what +Tillicum accepts: + +| klone (current) | Tillicum | why | +|---|---|---| +| `#SBATCH -p ckpt-g2` | **delete** | no partitions exist | +| `#SBATCH -q ckpt-gpu` | `#SBATCH --qos=normal` | or `long` once approved | +| `#SBATCH --requeue` | **delete** | nothing preempts | +| `#SBATCH --time=72:00:00` | `24:00:00` | 72 h needs `long` QoS | +| `#SBATCH --gpus-per-node=1` | `#SBATCH --gres=gpu:1` | also `--gpus=1` / `-G 1` | +| `#SBATCH --cpus-per-task=12` | **`8`** (1 GPU) | 12 CPUs on 1 GPU is rejected | +| `#SBATCH --mem=64G` | keep, or up to `200G` | no default; must be explicit | +| `#SBATCH --nodes=1 --ntasks=1` | unchanged | | + +**`--cpus-per-task=12` with one GPU is the blocking incompatibility** — it is not a +tuning preference, it exceeds the hard 8:1 ratio and the job will not run. + +Also drop the inherited mail settings. Our klone jobs carry +`MailType=END,FAIL,TIME_LIMIT` from a site default or the submit environment (it is +*not* in the script), which on a preemptable partition produced ~170 emails in one +night. On Tillicum there is no requeue storm to amplify it, but set `--mail-type=NONE` +explicitly so the behavior is the script's decision rather than the site's. + +## First runs: what to measure + +Spend the free demo hours on measurement, not on a production run. Three jobs, ~15 +GPU-hours total, answering the questions the #70 budget depends on: + +1. **`debug` QoS smoke test** (1 h, ~1 GPU-h) — does our environment run at all? + This is where the conda-vs-container question gets settled. +2. **One pano epoch, 1 GPU** (~1–3 h). klone L40S baseline: 2.6–2.8 h/epoch. An H200 + should be meaningfully faster, but *how much* is the number the budget needs. +3. **One tiles epoch, 1 GPU (8 CPU) vs 2 GPUs (16 CPU)** (~10 GPU-h). Settles the + 2-GPU trap above, and — more importantly — tells us whether tiles-vs-pano in the + #51 results is an **architecture finding or a storage artifact**. That question is + currently unresolved and it materially affects what the baseline means. + +Record the measured epoch times back into this file and into #70. + +## Admin + +- **Group:** `u_hyak_tillicum_makelab` (UW Groups Service). Jon is a member manager and + can add users via the Membership tab; UW-IT can grant member-manager rights to others + on request. +- **Maintenance:** second Tuesday monthly. Join the mailing list; put it on the run + calendar so long jobs are not scheduled across it. +- **Support:** `help@uw.edu` with "Tillicum" in the subject; twice-weekly office hours. +- **Budget changes:** contact UW-IT; enforcement semantics (blocks new submissions vs. + cancels running jobs) are **UNVERIFIED** and were asked about on 2026-07-30. + +## Answers from UW-IT (Sumaiya Sathar, 2026-07-30) + +All six questions answered. Several change the plan, so they are recorded here rather +than left in a mailbox. + +1. **Budget enforcement is safe.** Set to **$1,500/month, enforcement active**. Enforced + budgets **do not cancel running jobs** — a job already running continues to + completion; enforcement only **blocks new submissions** until the cap is raised or + the period rolls over. So the cap cannot destroy an in-flight experiment, which was + the only reason to have preferred warn-only. +2. **No CPUs without GPUs — confirmed, and it is a rate-model constraint, not a + scheduling one.** There is no QoS granting extra cores without the matching GPU + allocation; UW-IT is considering it, but the UW-approved rate model cannot charge for + CPUs independently, and unbalanced nodes are undesirable. **So the 2-GPU dataloader + trick is the only lever**, and its 2× billing is unavoidable — see the trap above. +3. **Many small files → `/gpfs/scrubbed`, and pack into SquashFS.** This is the big one. + Active many-small-file datasets belong on **`/gpfs/scrubbed`** (larger capacity, has + an automatic cleanup policy) rather than the 1 TB project quota, and UW-IT explicitly + recommends **SquashFS** for this shape. Data Commons is an option if the dataset can + be public. +4. **Storage grows in 1 TB increments** on request to `help@uw.edu` (subject "Tillicum") + with a workflow justification — it is shared active-compute storage, not archival. +5. **No project end date.** "Purged at the end of the project" simply means that if we + stop using Tillicum we must copy data off so the space can be reclaimed. Nothing + expires on a timer. +6. **1 TB is correct for us** — the 100 GB figure in the public docs applies to *demo* + accounts. Providing a worktag makes it a regular account, which is why we got 1 TB. +7. **The 100 free GPU hours do not expire.** They remain until used. +8. **`long` QoS requires the Special QoS Access Request Form** — still to be submitted, + then reviewed. Not a blocker for the measurement phase (everything fits in 24 h), but + it gates comfortable production runs. + +### What this changes + +- **Dataset goes to `/gpfs/scrubbed`, not `/gpfs/projects/makelab`.** The 1 TB project + quota is for artifacts we want backed up (checkpoints, results), not the 286 GB of + training data. This also removes storage pressure from the plan entirely. +- **Pack with SquashFS, not tar.** See + `scripts/model_comparison/pack_yolo_dataset.slurm`. tar fixes the *transfer* but not + the *destination* — untarring recreates all ~911k files on `/gpfs` and pays the + metadata cost permanently. A SquashFS image is mounted read-only and the destination + only ever holds one file. +- **Enforced budget is safe to leave on**, so the runaway-spend guard costs us nothing. +- **The `DEVICE=0` / 2-GPU design in the Tillicum launcher is now confirmed necessary** + rather than a workaround pending a better answer. + +## Related + +- #51 — supervised YOLO baseline (the runs that motivated this) +- #70 — stabilized rerun; **this is its unblocking dependency** +- `scripts/model_comparison/yolo_baseline/README.md` — preserved klone training record +- `hyak_yolo_runbook.sh`, `hyak_qwen_runbook.sh` — klone runbooks (repo root) diff --git a/scripts/model_comparison/pack_yolo_dataset.slurm b/scripts/model_comparison/pack_yolo_dataset.slurm new file mode 100644 index 0000000..c8127cc --- /dev/null +++ b/scripts/model_comparison/pack_yolo_dataset.slurm @@ -0,0 +1,123 @@ +#!/bin/bash +# Pack one YOLO dataset into a SquashFS image, for transfer klone -> Tillicum. +# +# WHY SQUASHFS AND NOT rsync/scp/tar +# The dataset is 286 GB in ~911,000 files (tiles 210 GB / 718,415; pano 76 GB / +# 192,938 -- measured 2026-07-30). That is metadata-bound, not bandwidth-bound: `du -sh` +# over the tiles tree beat three timeouts before finishing on a 550 s budget, and even +# `ls -lU | head -201` on tiles/images/train timed out. +# +# - rsync/scp of the tree pays ~911k round trips. No. +# - tar fixes the TRANSFER but not the DESTINATION: untarring recreates all 911k files +# on /gpfs, paying the metadata cost again and permanently. +# - SquashFS pays it ONCE, here, and the destination only ever sees ONE file. The +# training job mounts it read-only and the kernel serves the tree from a compact +# in-image index. This is also what UW-IT recommends for many-small-file datasets +# (Sumaiya, 2026-07-30). +# +# WHY A SLURM JOB AND NOT A LOGIN NODE +# klone reaps heavy login processes, and that reap also kills the SSH control master +# every agent session depends on. Never run this interactively. +# +# WHERE IT WRITES +# /gscratch/scrubbed, NOT /gscratch/makelab: lab storage is 84% full with ~168 GB free +# (checked 2026-07-30) and cannot hold this. scrubbed has ~15 TB free, and its ~21-day +# purge does not matter for a transient staging area. +# +# DS=pano sbatch -A ckpt-makelab scripts/model_comparison/pack_yolo_dataset.slurm +# DS=tiles sbatch -A ckpt-makelab scripts/model_comparison/pack_yolo_dataset.slurm +# +#SBATCH --job-name=yolo_pack +#SBATCH -p ckpt +#SBATCH -q ckpt +#SBATCH --requeue +#SBATCH --time=12:00:00 +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=8 +#SBATCH --mem=32G +#SBATCH --mail-type=NONE +#SBATCH --output=logs/yolo_pack_%j.out +#SBATCH --error=logs/yolo_pack_%j.err + +set -euo pipefail + +DS="${DS:?set DS to pano or tiles}" +SRC="${SRC:-/gscratch/scrubbed/$USER/yolo}" +OUT="${OUT:-/gscratch/scrubbed/$USER/yolo_squashfs}" +mkdir -p "$OUT" + +IMG="$OUT/${DS}.sqfs" +SUM="$IMG.sha256" + +echo "--- pack $DS ---" +echo "src: $SRC/$DS out: $IMG" +df -h "$OUT" | tail -1 + +# Idempotent: a completed, verified image is left alone, so a ckpt requeue or a manual +# resubmit is free rather than a restart from zero. +if [ -f "$IMG" ] && [ -f "$SUM" ]; then + echo "[skip] image exists; verifying..." + if ( cd "$OUT" && sha256sum -c "$(basename "$SUM")" ); then + echo "[skip] verified, nothing to do"; exit 0 + fi + echo "[warn] checksum FAILED -- rebuilding" + rm -f "$IMG" "$SUM" +fi + +# --------------------------------------------------------------------------------- +# THE ULTRALYTICS LABEL CACHE IS THE SUBTLE PART -- READ BEFORE CHANGING THIS. +# +# Ultralytics writes labels/.cache next to the labels and validates it against a +# hash of the label+image FILE PATHS. Two consequences: +# +# 1. The .cache files MUST be inside the image. They already exist on klone (built +# 2026-07-25/26), and a read-only mount cannot create them. Without them every run +# rescans ~911k files -- exactly the cost this whole exercise exists to avoid. +# +# 2. The hash covers absolute paths, so mounting at a DIFFERENT path than the one the +# cache was built under INVALIDATES it and forces that same rescan. Mount the image +# on Tillicum at a stable path and expect to regenerate the cache ONCE there (it +# needs a writable overlay, or a one-off run with labels/ bind-mounted writable). +# Budget one slow first epoch; do not mistake it for the steady-state cost. +# +# This is called out in docs/tillicum.md too, because getting it wrong is invisible -- +# training still works, it is just silently slow forever. +# --------------------------------------------------------------------------------- +echo "[check] label caches that will be packed:" +ls -la "$SRC/$DS/labels/"*.cache 2>/dev/null || echo " NONE -- expect a full rescan on first use" + +# -noD: do NOT compress data blocks. The payload is JPEG, already compressed; deflating +# it would burn hours of CPU for ~nothing and make reads slower. Metadata/inodes are +# still compressed, which is where the win actually is. +# -no-xattrs: nothing here uses them, and they bloat the index. +echo "[mksquashfs] starting $(date)" +mksquashfs "$SRC/$DS" "$IMG" \ + -noD \ + -no-xattrs \ + -processors "${SLURM_CPUS_PER_TASK:-8}" \ + -info +echo "[mksquashfs] done $(date)" + +# The checksum travels with the image and is verified after transfer, same discipline as +# the weight snapshot's MANIFEST.md. +( cd "$OUT" && sha256sum "$(basename "$IMG")" > "$(basename "$SUM")" ) + +ls -lh "$IMG" +cat "$SUM" + +cat < + +Verify the checksum on arrival BEFORE trusting it: + sha256sum -c $(basename "$SUM") +EOF diff --git a/scripts/model_comparison/run_yolo_data_prep_tillicum.slurm b/scripts/model_comparison/run_yolo_data_prep_tillicum.slurm new file mode 100644 index 0000000..92c471b --- /dev/null +++ b/scripts/model_comparison/run_yolo_data_prep_tillicum.slurm @@ -0,0 +1,92 @@ +#!/bin/bash +# Rebuild the YOLO training dataset ON TILLICUM, from Hugging Face. Download + prep. +# +# WHY REGENERATE RATHER THAN TRANSFER FROM KLONE +# There is no automated path from klone to Tillicum: no shared filesystem (/gscratch and +# /mmfs1 do not exist here), and Tillicum cannot authenticate to klone non-interactively +# — both ends are Duo/keyboard-interactive with no publickey, so BatchMode fails. Globus +# CLI is on neither side. Verified 2026-07-30. +# +# Regenerating sidesteps all of it, and is SAFE because the prep is deterministic by +# construction: prepare_yolo_dataset.py thins background tiles with an md5 of the file +# stem specifically so the choice is stable across processes and runs (see +# `_keep_background`, and its comment about salted hash()). +# +# VERIFY, DO NOT ASSUME. The committed record gives the exact counts klone produced: +# tiles 557,413 train / 161,002 val pano 150,063 train / 42,875 val +# This job prints its own counts at the end. If they do not match, the datasets are NOT +# equivalent and nothing trained here is comparable to the #51 runs — stop and diff. +# +# WHY A GPU JOB FOR CPU WORK +# Tillicum prohibits CPU-only jobs; every job must request >=1 GPU. The runbook also +# warns that login nodes reap heavy long-running processes, and on Tillicum that would +# take the Duo-authenticated SSH master with it. So we hold one GPU and eat the cost: +# roughly 6-12 h at $0.90/h, i.e. $5-11. Cheap against a $1,500/month cap. +# +# sbatch scripts/model_comparison/run_yolo_data_prep_tillicum.slurm +# +#SBATCH --job-name=yolo_data_prep +#SBATCH --qos=normal +#SBATCH --gres=gpu:1 +#SBATCH --cpus-per-task=8 +#SBATCH --mem=200G +#SBATCH --time=12:00:00 +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --mail-type=NONE +#SBATCH --output=logs/yolo_data_prep_%j.out +#SBATCH --error=logs/yolo_data_prep_%j.err + +set -euo pipefail + +export REPO="${REPO:-/gpfs/home/$USER/RampNet}" +# /gpfs/scrubbed, NOT the 1 TB /gpfs/projects/makelab quota: the HF source is ~462 GB +# and the prepared YOLO layout another ~286 GB, so this needs ~750 GB. scrubbed has +# ~484 TB free and is what UW-IT recommends for active many-small-file datasets. +export SCRATCH="${SCRATCH:-/gpfs/scrubbed/$USER}" +export PYBIN="${PYBIN:-/gpfs/projects/makelab/$USER/envs/rampnet-yolo/bin/python}" +export HF_HOME="$SCRATCH/hf" +export YOLO_CONFIG_DIR="$SCRATCH/ultralytics" + +mkdir -p "$SCRATCH" + +echo "=== node ===" +hostname; date +df -h "$SCRATCH" | tail -1 +echo "=== interpreter (must match the klone #51 runs exactly) ===" +"$PYBIN" -c "import sys, torch, ultralytics; print('python', sys.version.split()[0], '| torch', torch.__version__, '| ultralytics', ultralytics.__version__)" + +cd "$REPO" + +echo +echo "=== STAGE: data (HF download, ~462 GB) ===" +date +bash hyak_yolo_runbook.sh data + +echo +echo "=== STAGE: prep (tiles + pano) ===" +date +bash hyak_yolo_runbook.sh prep + +echo +echo "=== VERIFICATION — counts must match the klone record ===" +date +for ds in tiles pano; do + for split in train val; do + d="$SCRATCH/yolo/$ds/images/$split" + printf " %-6s %-6s %s\n" "$ds" "$split" "$(ls -U "$d" 2>/dev/null | grep -c '\.jpg$')" + done +done +cat <<'EOF' + + Expected, from scripts/model_comparison/yolo_baseline/README.md provenance: + tiles train 557413 + tiles val 161002 + pano train 150063 + pano val 42875 + + A mismatch means the regenerated dataset is NOT the one the #51 arms trained on. + Do not train against it until the difference is understood. +EOF +date +echo "Cost so far: run `hyakusage`" diff --git a/scripts/model_comparison/run_yolo_train_tillicum.slurm b/scripts/model_comparison/run_yolo_train_tillicum.slurm new file mode 100644 index 0000000..08feaf5 --- /dev/null +++ b/scripts/model_comparison/run_yolo_train_tillicum.slurm @@ -0,0 +1,161 @@ +#!/bin/bash +# Train one YOLO curb-ramp detector on the RampNet dataset, on TILLICUM. +# +# Port of run_yolo_train.slurm (klone). Kept as a SEPARATE file rather than +# branching inside the klone script, because nearly every scheduler directive +# differs and the klone version is the preserved record of the #51 runs -- it +# should not churn. See docs/tillicum.md for the directive-by-directive diff. +# +# WHAT CHANGED FROM THE KLONE VERSION, AND WHY +# -p ckpt-g2 / -q ckpt-gpu -> --qos=normal Tillicum has NO partitions; QoS only. +# --requeue -> dropped Nothing preempts on Tillicum. +# --time=72:00:00 -> 24:00:00 normal QoS caps at 24 h. 7-day jobs +# need `long` QoS (pre-approval). +# --gpus-per-node=1 -> --gres=gpu:N Tillicum's documented spelling. +# --cpus-per-task=12 -> 8 x N GPUs HARD 8:1 ratio; 12-on-1 is REJECTED. +# (no mail directive) -> --mail-type=NONE Be explicit so the site default +# cannot decide for us (on klone an +# inherited END,FAIL,TIME_LIMIT produced +# ~1 email per minute under preemption). +# +# YOLO_DATA=$DATA/pano/data.yaml YOLO_IMGSZ=1280 BATCH=4 NAME=y26_pano_till \ +# sbatch scripts/model_comparison/run_yolo_train_tillicum.slurm +# +# COST. Tillicum BILLS: $0.90/GPU-hour where GPU-hour = elapsed x N GPUs. A 24 h +# 2-GPU job is 48 GPU-hours (~$43). Do not leave allocations idle. Check `hyakusage`. +#SBATCH --qos=normal +#SBATCH --job-name=yolo_curb_ramp_train +#SBATCH --time=24:00:00 +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --mail-type=NONE +#SBATCH --output=logs/yolo_train_till_%j.out +#SBATCH --error=logs/yolo_train_till_%j.err +# gres/cpus/mem are NOT hardcoded -- see the GPUS block below, which must be kept in +# step with them. Override at submit time with: sbatch --gres=gpu:2 --cpus-per-task=16 +#SBATCH --gres=gpu:1 +#SBATCH --cpus-per-task=8 +#SBATCH --mem=200G + +set -euo pipefail + +# --------------------------------------------------------------------------------- +# The 2-GPU dataloader trick, and why DEVICE is separate from the allocation. +# +# Tillicum fixes CPUs at 8 per GPU. Our tiles arm is I/O-bound (the same epoch took +# 4.1-9.5 h on klone depending only on which node it landed on), so more dataloader +# workers is the obvious lever -- but on Tillicum more CPUs means more GPUs. +# +# The naive move -- allocate 2 GPUs and let ultralytics use both -- is NOT a clean +# experiment: 2 GPUs puts ultralytics into DDP, which changes the effective batch and +# the LR dynamics, so the result is no longer comparable to the klone single-GPU runs. +# +# So: allocate 2 GPUs for their 16 CPUs, but pin DEVICE=0 so training math is +# IDENTICAL to a 1-GPU run. The second GPU sits idle and we pay for it. That is the +# point -- it isolates dataloader throughput as the only changed variable, and it +# answers whether the 2x bill buys more than a 2x speedup. Set DEVICE=0,1 only if you +# deliberately want a DDP run, and label it as a different arm. +# --------------------------------------------------------------------------------- +DEVICE="${DEVICE:-0}" + +YOLO_CKPT="${YOLO_CKPT:-yolo11l.pt}" +YOLO_DATA="${YOLO_DATA:?set YOLO_DATA to a prepared data.yaml}" +YOLO_IMGSZ="${YOLO_IMGSZ:-1024}" +EPOCHS="${EPOCHS:-60}" +BATCH="${BATCH:--1}" # pin per config to match the klone runs +PATIENCE="${PATIENCE:-20}" + +# Keep a checkpoint every N epochs. Ultralytics defaults save_period=-1, keeping ONLY +# last.pt and best.pt -- which forecloses, permanently and retroactively, any analysis +# that needs a checkpoint from a specific epoch. +# +# The one we care about: RampNet's published model was trained for ~1 epoch / ~9.4k +# steps at constant LR (#84), while this baseline gets 60 epochs with a schedule. If +# RampNet wins anyway that is a STRONGER result -- but only if we can also report a +# COMPUTE-MATCHED YOLO point, i.e. the checkpoint at the epoch where YOLO has consumed +# comparable training, not just the converged one. That requires the checkpoint to +# still exist. results.csv gives per-epoch val metrics but no weights, so it cannot be +# reconstructed after the fact. +# +# Cost is disk (~150 MB per checkpoint per arm) against a 1 TB allocation. Cheap +# insurance for an option that cannot be bought back later. +SAVE_PERIOD="${SAVE_PERIOD:-5}" +# On klone this defaulted to 0 because ultralytics `time=` restarts its clock on every +# preemption. Tillicum never preempts, so `time=` is safe here -- but leave it 0 for +# any run meant to be comparable to the #51 klone results, which are epoch-based. +TRAIN_HOURS="${TRAIN_HOURS:-0}" +NAME="${NAME:-yolo_run_till}" + +# Tillicum storage. /gpfs/projects/makelab is 1 TB, backed up daily, and PURGED AT END +# OF PROJECT -- it is NOT archival. Only best.pt matters downstream: rsync it back to +# klone /gscratch/makelab, which stays the system of record. +GPFS="${GPFS:-/gpfs/projects/makelab/$USER}" +export HF_HOME="${HF_HOME:-$GPFS/hf}" +PROJECT="${PROJECT:-$GPFS/yolo_runs}" +export YOLO_CONFIG_DIR="${YOLO_CONFIG_DIR:-$GPFS/ultralytics}" +mkdir -p "$PROJECT" "$YOLO_CONFIG_DIR" "$HF_HOME" + +# PYTHON points at an interpreter with ultralytics. On Tillicum the DOCUMENTED path is +# an Apptainer container (NGC PyTorch + `pip install ultralytics`), not a conda module +# -- our environment.yml pins CUDA 11.8 against what is now a Rocky 9 / H200 host and +# should not be assumed to transfer. Set PYTHON, or set APPTAINER_IMG to run in one. +PYTHON="${PYTHON:-python}" +APPTAINER_IMG="${APPTAINER_IMG:-}" + +echo "--- YOLO train on TILLICUM (issue #51 / #70) ---" +echo "base: ${YOLO_CKPT}" +echo "data: ${YOLO_DATA}" +echo "imgsz: ${YOLO_IMGSZ} epochs: ${EPOCHS} batch: ${BATCH} patience: ${PATIENCE}" +echo "alloc: ${SLURM_GPUS_ON_NODE:-?} GPU(s), ${SLURM_CPUS_PER_TASK:-?} CPUs on ${SLURMD_NODENAME:-?}" +echo "device: ${DEVICE} (allocation and device differ on purpose -- see header)" +echo "out: ${PROJECT}/${NAME}/weights/best.pt" +nvidia-smi --query-gpu=name,memory.total --format=csv || true +echo "------------------------------------------------" + +run_train() { + "$@" - "$YOLO_CKPT" "$YOLO_DATA" "$YOLO_IMGSZ" "$EPOCHS" "$BATCH" "$PATIENCE" \ + "$PROJECT" "$NAME" "$TRAIN_HOURS" "$DEVICE" "$SAVE_PERIOD" <<'PY' +import os, sys +from ultralytics import YOLO +(ckpt, data, imgsz, epochs, batch, patience, project, name, hours, device, + save_period) = sys.argv[1:12] +last = os.path.join(project, name, "weights", "last.pt") +# Still needed on Tillicum, but for a different reason than on klone: not preemption, +# but the 24 h normal-QoS ceiling. A 60-epoch tiles schedule does not fit in one job, +# so chain jobs and resume. Unlike klone the boundary is predictable, not random. +# +# CAVEAT: resume=True reuses every training arg SAVED IN THE CHECKPOINT, so a run +# resumed from a klone checkpoint keeps that run's save_period=-1 -- SAVE_PERIOD above +# does NOT apply to it. Per-epoch checkpoints therefore exist only for runs STARTED +# under this script. For the klone arms the early-epoch weights are already gone. +# +# That is less costly than it sounds: the compute-matched point we most want (YOLO at +# roughly RampNet's ~1-epoch budget) is exactly the ep1 best.pt that four of the six +# klone arms are still holding. The "one-epoch models" are the matched comparison. +if os.path.exists(last): + print(f"[resume] {last} exists -> resuming", flush=True) + YOLO(last).train(resume=True) +else: + bf = float(batch) + batch_arg = int(bf) if bf.is_integer() else bf + dev = [int(d) for d in device.split(",")] if "," in device else int(device) + kw = dict( + data=data, imgsz=int(imgsz), epochs=int(epochs), batch=batch_arg, + patience=int(patience), project=project, name=name, device=dev, exist_ok=True, + save_period=int(save_period), + ) + if float(hours) > 0: + kw["time"] = float(hours) + YOLO(ckpt).train(**kw) +PY +} + +# run_train takes the interpreter as "$@", so the container case is just a longer +# command word -- the heredoc still lands on its stdin and every shell variable is +# expanded HERE, outside the container, before apptainer is ever invoked. +# --nv exposes the GPUs; bind /gpfs so the container sees data, runs and cache. +if [ -n "$APPTAINER_IMG" ]; then + run_train apptainer exec --nv --bind /gpfs:/gpfs "$APPTAINER_IMG" python +else + run_train "$PYTHON" +fi diff --git a/scripts/tillicum_recon.sh b/scripts/tillicum_recon.sh new file mode 100644 index 0000000..de640dc --- /dev/null +++ b/scripts/tillicum_recon.sh @@ -0,0 +1,83 @@ +#!/bin/bash +# Tillicum first-login recon. Read-only, ~10 seconds, submits nothing and costs nothing. +# +# Everything in docs/tillicum.md marked UNVERIFIED was written from the public docs +# without an account. This settles those items in one pass so the doc, the dotfiles +# ssh target, and run_yolo_train_tillicum.slurm can be corrected from fact. +# +# ./scripts/tillicum_recon.sh (on a Tillicum login node) +# wsl-ssh.ps1 tillicum script scripts/tillicum_recon.sh (from a Claude session) +# +# CPU-only jobs are prohibited on Tillicum, so this deliberately runs on the LOGIN +# node -- it is all metadata queries, nothing that would warrant an allocation. + +echo "==============================================================" +echo " Tillicum recon -- $(date)" +echo " host: $(hostname) user: $USER" +echo "==============================================================" + +echo +echo "## 1. HOME PATH (the top unknown -- wsl-ssh.ps1 currently guesses)" +echo "home: $(cd ~ && pwd)" +echo "quota:" +( quota -s 2>/dev/null || echo " (quota not available)" ) | sed 's/^/ /' + +echo +echo "## 2. PROJECT STORAGE" +for d in /gpfs/projects/makelab /gpfs/projects /gpfs; do + if [ -d "$d" ]; then + echo "$d exists; writable=$( [ -w "$d" ] && echo yes || echo NO )" + df -h "$d" 2>/dev/null | tail -1 | sed 's/^/ /' + break + else + echo "$d MISSING" + fi +done +echo "contents of /gpfs/projects/makelab:" +ls -la /gpfs/projects/makelab 2>&1 | head -10 | sed 's/^/ /' + +echo +echo "## 3. SCHEDULER (expect: no partitions, QoS-driven)" +echo "-- sinfo --" +sinfo -o "%.14P %.6a %.10l %.6D %.6t %N" 2>&1 | head -10 | sed 's/^/ /' +echo "-- QoS available to us --" +sacctmgr -nP show assoc user="$USER" format=Account,QOS 2>&1 | sed 's/^/ /' +echo "-- do we already have long/wide/urgent? --" +sacctmgr -nP show assoc user="$USER" format=QOS 2>/dev/null | tr ',' '\n' \ + | grep -iE "long|wide|urgent" | sed 's/^/ /' || echo " (none -- long QoS needs the request form)" + +echo +echo "## 4. BILLING" +echo "-- hyakusage --" +( command -v hyakusage >/dev/null && hyakusage 2>&1 || echo " hyakusage NOT on PATH" ) | sed 's/^/ /' + +echo +echo "## 5. SOFTWARE (docs say Apptainer, not modules -- verify)" +for t in apptainer singularity module conda python3 nvidia-smi rsync globus; do + p=$(command -v "$t" 2>/dev/null) + printf " %-12s %s\n" "$t" "${p:-NOT FOUND}" +done +echo "-- apptainer version --" +( apptainer --version 2>&1 || echo " n/a" ) | sed 's/^/ /' +echo "-- modules, if any --" +( module avail 2>&1 | head -15 || echo " n/a" ) | sed 's/^/ /' + +echo +echo "## 6. MAIL (klone forced END,FAIL,TIME_LIMIT via a lua job_submit plugin and" +echo "## ~1 email/min under preemption -- check whether Tillicum does the same)" +scontrol show config 2>/dev/null \ + | grep -iE "JobSubmitPlugins|MailProg|MailDomain|PreemptMode|MaxJobCount" | sed 's/^/ /' + +echo +echo "## 7. NETWORK PATH BACK TO KLONE (286 GB / ~911k files has to cross)" +echo " NOTE: transfer ARCHIVES, not the tree. Per-file copy would pay ~911k round trips." +for h in klone.hyak.uw.edu; do + printf " %-24s " "$h" + ( timeout 5 bash -c "/dev/null && echo "port 22 reachable" ) || echo "port 22 NOT reachable from here" +done + +echo +echo "==============================================================" +echo " Next: update docs/tillicum.md (home path, QoS, mail, software)," +echo " fix Home in wsl-ssh.ps1 \$Targets, then run the debug smoke test." +echo "==============================================================" diff --git a/scripts/tillicum_setup_env.sh b/scripts/tillicum_setup_env.sh new file mode 100644 index 0000000..f1dbd2c --- /dev/null +++ b/scripts/tillicum_setup_env.sh @@ -0,0 +1,91 @@ +#!/bin/bash +# Build a Tillicum training environment that MATCHES the klone #51 runs exactly. +# +# WHY EXACT PINS AND NOT "latest" +# The whole point of the Tillicum move is to finish the supervised-YOLO baseline so it +# can be compared against RampNet without a "you undertrained the baseline" objection. +# Resuming klone checkpoints into a different torch/ultralytics, or comparing a +# Tillicum-trained arm against a klone-trained one across library versions, reintroduces +# exactly the uncontrolled variable that #71's protocol exists to eliminate. +# +# The reference is not the README -- it is the training job logs themselves +# (logs/yolo_train_37745363.out on klone), which report: +# +# Python-3.11.15 Ultralytics 8.4.105 torch-2.13.0+cu126 +# +# A naive `pip install torch ultralytics` on Tillicum gives torch 2.8.0 + ultralytics +# 8.4.113, because Tillicum's SYSTEM python is 3.9.25 and the cu126 index tops out at +# 2.8.0 for 3.9. Hence the conda module: it is the only way to get 3.11 here. +# +# WHERE IT LIVES +# /gpfs/projects/makelab -- the 1 TB backed-up allocation, not /gpfs/scrubbed. The +# dataset belongs on scrubbed (it is huge and reproducible); the environment is small +# and annoying to rebuild, so it belongs where there are backups. +# +# wsl-ssh.ps1 tillicum script scripts/tillicum_setup_env.sh + +set -euo pipefail + +PYVER="3.11.15" +TORCH="2.13.0+cu126" +ULTRA="8.4.105" +TORCH_INDEX="https://download.pytorch.org/whl/cu126" + +ENVROOT="${ENVROOT:-/gpfs/projects/makelab/$USER/envs}" +ENVDIR="$ENVROOT/rampnet-yolo" + +echo "=== target: python $PYVER / torch $TORCH / ultralytics $ULTRA ===" +mkdir -p "$ENVROOT" + +module load conda +conda --version + +if [ ! -x "$ENVDIR/bin/python" ]; then + echo "=== creating conda env at $ENVDIR ===" + conda create -y -p "$ENVDIR" "python=$PYVER" +else + echo "=== env exists, reusing $ENVDIR ===" +fi + +PY="$ENVDIR/bin/python" +"$PY" -V + +echo "=== is the pinned torch actually available for this interpreter? ===" +# Fail loudly HERE rather than silently installing a different version: a mismatched +# torch is the kind of thing that produces plausible numbers and an unpublishable +# comparison. +if ! "$PY" -m pip index versions torch --index-url "$TORCH_INDEX" 2>&1 | grep -q "${TORCH}"; then + echo "!! torch $TORCH NOT available for $("$PY" -V). Available:" + "$PY" -m pip index versions torch --index-url "$TORCH_INDEX" 2>&1 | head -3 + echo "!! Refusing to install a substitute -- that would break comparability with #51." + exit 1 +fi + +echo "=== installing pinned torch ===" +"$PY" -m pip install --upgrade pip +"$PY" -m pip install "torch==$TORCH" --index-url "$TORCH_INDEX" + +echo "=== installing pinned ultralytics ===" +# --no-deps would risk a broken install; instead pin ultralytics and let it resolve its +# own deps, then assert torch was not silently upgraded underneath us. +"$PY" -m pip install "ultralytics==$ULTRA" + +echo "=== VERIFY (this is the gate, not the install) ===" +"$PY" - </dev/null | tail -1 | awk '{printf "size=%s avail=%s\n", $2, $4}' + else + printf "%-32s MISSING\n" "$d" + fi +done + +say "SQUASHFS SUPPORT — can we mount the packed dataset at all?" +for t in mksquashfs unsquashfs squashfuse apptainer; do + printf " %-12s %s\n" "$t" "$(command -v $t 2>/dev/null || echo NOT_FOUND)" +done +echo " kernel squashfs: $(grep -qw squashfs /proc/filesystems && echo yes || echo 'no (use apptainer --bind image-src=/)')" + +say "CONDA MODULE PATH" +module load conda 2>&1 | head -5 +command -v conda && conda --version + +# Build somewhere durable and shared, not node-local /tmp. Keep it out of the 286 GB +# dataset's way: envs are small but numerous-filed, so scrubbed is the better home. +ENVROOT="${ENVROOT:-/gpfs/scrubbed/$USER/envs}" +if ! mkdir -p "$ENVROOT" 2>/dev/null; then + echo " (could not write $ENVROOT -- falling back to the project dir)" + ENVROOT="/gpfs/projects/makelab/$USER/envs" + mkdir -p "$ENVROOT" +fi +VENV="$ENVROOT/ultra" +echo "env root: $ENVROOT" + +say "BUILD A MINIMAL ULTRALYTICS ENV" +# A venv on top of the module python is far lighter than solving environment.yml, and +# the only question that matters here is whether a CUDA-12 torch wheel drives an H200. +if [ ! -x "$VENV/bin/python" ]; then + python3 -m venv "$VENV" 2>&1 | tail -2 +fi +"$VENV/bin/python" -m pip install -q --upgrade pip 2>&1 | tail -2 +# cu126 wheels: Tillicum is sm_90, and our klone stack is torch 2.13.0+cu126 (see the +# yolo_baseline record), so matching it keeps the comparison honest. +"$VENV/bin/python" -m pip install -q torch --index-url https://download.pytorch.org/whl/cu126 2>&1 | tail -3 +"$VENV/bin/python" -m pip install -q ultralytics 2>&1 | tail -3 + +say "DOES IT SEE THE GPU?" +"$VENV/bin/python" - <<'PY' +import sys +try: + import torch + print("torch:", torch.__version__, "| cuda build:", torch.version.cuda) + print("cuda available:", torch.cuda.is_available()) + if torch.cuda.is_available(): + print("device:", torch.cuda.get_device_name(0)) + print("capability:", torch.cuda.get_device_capability(0), "(expect (9,0) for H200)") + import time + a = torch.randn(8192, 8192, device="cuda", dtype=torch.bfloat16) + torch.cuda.synchronize(); t0 = time.time() + for _ in range(50): + a @ a + torch.cuda.synchronize() + dt = time.time() - t0 + tflops = 50 * 2 * 8192**3 / dt / 1e12 + print(f"bf16 matmul: {tflops:.0f} TFLOP/s ({dt:.2f}s for 50 iters)") + print(" ref: L40S ~180 dense. A large ratio here is the H200 speedup we are betting on.") +except Exception as e: + print("TORCH FAILED:", type(e).__name__, e); sys.exit(0) + +try: + from ultralytics import YOLO + import ultralytics + print("ultralytics:", ultralytics.__version__) + m = YOLO("yolo11n.yaml") # from config, no download needed + print("model constructed OK") +except Exception as e: + print("ULTRALYTICS FAILED:", type(e).__name__, e) +PY + +say "SLURM VIEW OF THIS JOB (confirm debug really is free)" +scontrol show job "$SLURM_JOB_ID" | grep -oE "QOS=[a-z]+|Partition=[a-z0-9-]+|TRES=[^ ]+" | head -5 + +say "DONE" +date +echo "Check cost with: hyakusage (this job should add 0.00)"